Data engineering in New York
New York's ad-tech and media companies produce event data faster than almost anyone in the city: impressions, clicks, views and conversions arriving from several vendors, each with its own clock and its own idea of what counts. Every client report is a reconciliation between those sources, and a campaign that under-delivers by a small margin becomes a make-good or a dispute.
We build the pipeline so the numbers agree, or show exactly where they do not. Raw events land untouched, so any figure can be rebuilt. Deduplication and time zones are handled once, in the pipeline, not separately in each dashboard. ClickHouse suits this volume, and we run it in production. Every report carries a freshness statement and a check that proves it, because an account manager quoting yesterday's delivery as today's is how a client stops trusting the whole dashboard.
Most software bought in New York is bought by companies that do not think of themselves as software companies. Banks, insurers, asset managers, publishers, agencies, fashion houses and property firms all run on systems that sit beside the real business: a client portal, a pricing engine, a reconciliation job, a content pipeline. Those systems rarely start from nothing. They connect to a vendor platform, a core system older than the team, and a folder of spreadsheets someone updates by hand, and the brief is usually to make all three agree.
What those companies need built follows the industry. Asset managers and insurers need client portals where a family office sees two funds and not a third, and overnight jobs that reconcile custodian, market data and order feeds before the market opens. Publishers and ad-tech firms need event pipelines whose delivery numbers agree across vendors, and front ends that stay fast under the ad stack. Property firms need apps for technicians working in basements with no signal. And founders who left one of these industries need a first version in front of a pilot customer before the interest cools.
Hiring is the constraint. Engineers here are courted by banks, big tech offices and funded start-ups at the same time, and Built In puts the average software engineer base salary in the city at about $160,000 before bonus. The shape this produces is a small in-house team that keeps the core systems running, and behind it a list of well-defined projects that nobody on that team will reach this year. That list is where we are useful: each project is a defined piece of work, it starts with a fixed-price two-week piece, and the code belongs to you from the first commit.
We are in Bengaluru and move our working day for New York. Four hours of every working day overlap with your morning in Eastern time, stand-up included, so decisions and code review happen live with the engineer who writes the code. Runbooks and architecture notes are written as we go, so your in-house team can run what we build long after the project is finished.
We are the right fit for the defined projects your in-house team is too stretched to reach this year: a client portal, a reconciliation job, an event pipeline, a first version for a pilot customer. Senior capacity starts within days, the first piece is a fixed-price two-week engagement, and agencies can ship the work under their own name.
Four ways this arrives.
Two definitions of the same metric, computed in two places. We pick one, write it as a tested model in version control, and point every report at it, so the argument moves from whose number is right to what the definition should be.
A job reports success while the report still shows yesterday. We give every table a written freshness guarantee, monitor what the reader actually sees rather than the job that feeds it, and alert a person when the guarantee is missed.
A CRM, an ERP or a work-order tool reachable only through its API. We build the sync with retries, rate limits and normalisation at the boundary, and keep append-only snapshots so last Tuesday is a query rather than a restore.
Analytical queries running against the application database at the worst possible moment. We move them to a columnar store, keep Postgres for transactions, and feed one from the other by change data capture or scheduled extracts, whichever the freshness requirement calls for.
A nightly work-order extract into append-only ClickHouse snapshots, with a verifier that reads the live Tableau dashboard to prove it shows the newest data.
Read the write-up →Asked by New York teams.
How do you work with teams in New York?+
We are in Bengaluru and move our working day for New York, so four hours of every working day overlap with your morning in Eastern time, stand-up included. Calls, reviews and decisions happen in that window, and the rest of the conversation runs in your tools: Slack, GitHub, Linear. The person on every call is the engineer who writes your code, and the code sits in your repositories from the first commit.
Do you build for New York financial firms?+
Yes. The work is usually the systems beside the trading or underwriting desk: overnight jobs that reconcile custodian, market data and order feeds before the open, client portals where investors, brokers and their accountants each see exactly what they should, and internal tools that replace a pricing or renewals workbook only one person understands. We build them to be idempotent and traceable, so every figure can be followed back to the file it came from.
How does your rate compare to hiring in New York?+
Built In puts the average software engineer base salary in New York City at about $160,000, before bonus and benefits. Our published rate is $35/hour, or $5,400 a month for an embedded engineer, with a $5,000 minimum. There is no recruiting time and no employment overhead, and work starts within days of the call. The first piece is a fixed-price two-week engagement at $2,800, so you judge us on working code.
Can you take over a system another vendor built?+
Yes. In New York that is often a client portal or an internal tool built by an agency that has since moved on, with nobody in-house who knows how it runs. We start by reading the code and running it, then write down how it works and where it is fragile before changing anything. A good first two-week piece makes the riskiest part safe and leaves your team runbooks it can use.
Do we need a data warehouse yet?+
Maybe not. If your reporting runs comfortably against a Postgres read replica, a warehouse adds a second system to pay for and keep in step. The signs you need one: analytical queries slowing the product, data from several sources that has to be joined, or history you must keep that the application overwrites. We will tell you which side of that line you are on.
Why ClickHouse rather than Snowflake or BigQuery?+
Not always. For steady, high-volume analytical work, ClickHouse is fast and cheap to run, self-hosted or on its managed cloud, and we run it in production. Snowflake and BigQuery ask less of you operationally and suit ad hoc querying across a large team. If nobody on your side wants to operate a database, we will recommend the managed option even when the invoice is larger.
What does a freshness guarantee actually mean?+
A written statement per table or report, such as "never more than an hour behind" or "reflects yesterday's close by 07:00". Each one has a check that measures it and an alert that fires when it is missed. Without it, stale data looks exactly like correct data until somebody makes a decision on it.
Can you work with the tools we already have?+
Usually, yes. If you already run dbt, Airflow, Fivetran or a BI tool your team knows, we build inside it. We suggest replacing a tool only when it is the cause of the problem you hired us for, and we show you the evidence before asking you to fund a migration.
How do you handle personal data in the warehouse?+
We decide what should reach the warehouse at all before building the pipeline. Identifiers can be hashed or tokenised at ingestion, sensitive columns restricted by role, and retention enforced by the pipeline itself. Every field is documented from source to dashboard, so anyone can see where a number came from and who can read it.