Data engineering in Chicago
At the end of every trading day, a Chicago firm has to agree with its clearing firm about positions, trades and cash, and the files that arrive to prove it rarely match the internal view on the first pass. The latency-critical trading path stays with specialists. The pipeline that turns the day's activity into reconciled positions, risk reports and a clean history is data engineering, and that is the part we build.
We ingest broker, clearing and internal data on a schedule, keep append-only snapshots so any past day can be reconstructed exactly, and surface breaks as a report someone owns rather than a number that quietly differs. For analytical volumes we use ClickHouse, as we do on FM360, with checks that prove each report is built from the latest data before anyone reads it.
Chicago's software buyers are mostly not startups selling to other startups. The city's economy rests on trading and derivatives, around CME Group and the proprietary trading firms that grew up beside the exchanges; on freight and logistics, because this is where much of the country's rail and trucking traffic meets; on insurance, from carriers to the brokers and agencies around them; and on manufacturing and distribution across the Midwest. These are established, profitable businesses, and a good deal of their software is older than some of the engineers who maintain it.
So the work is integration and modernisation more often than a blank page. Freight systems speak EDI and carrier APIs that disagree about what happened to a load. Insurers run on policy administration systems nobody wants to replace and everybody wants to build around. Trading firms keep their latency-critical code with in-house specialists, and buy the layer around it: risk and reconciliation reports, operations dashboards, internal tools and data pipelines. The strongest pitch in Chicago is rarely a rewrite. It is a clean, documented system built beside the old one, with work moved across one piece at a time.
Built In puts the average base salary for a software engineer in Chicago at around $133,000, and every employer in the city competes with the trading firms for the same engineers. The result is a familiar backlog: the carrier integration that has to be done properly once, the agency portal everyone complains about, the internal tools the engineering team is too busy to reach. Those are defined pieces of work, which suits an engagement. It starts with a fixed-price two-week piece, and the code belongs to you from the first commit.
We move our working day for Illinois. Four hours of every working day overlap with your morning in Central time, stand-up included, so calls, reviews and decisions happen live with the engineer who writes the code. Everything decided or discovered is written down in your repository and tracker, and runbooks and architecture notes are written as we go, so the system built beside the old one is documented from its first day.
We are the right fit for a defined piece of modernisation, an integration that has to be done properly once, or the internal tools your own engineers are too busy to reach. Senior capacity starts within days, the first piece is a fixed-price two-week engagement, and agencies serving Chicago's insurers can put the work under their own name.
Four ways this arrives.
Two definitions of the same metric, computed in two places. We pick one, write it as a tested model in version control, and point every report at it, so the argument moves from whose number is right to what the definition should be.
A job reports success while the report still shows yesterday. We give every table a written freshness guarantee, monitor what the reader actually sees rather than the job that feeds it, and alert a person when the guarantee is missed.
A CRM, an ERP or a work-order tool reachable only through its API. We build the sync with retries, rate limits and normalisation at the boundary, and keep append-only snapshots so last Tuesday is a query rather than a restore.
Analytical queries running against the application database at the worst possible moment. We move them to a columnar store, keep Postgres for transactions, and feed one from the other by change data capture or scheduled extracts, whichever the freshness requirement calls for.
A nightly work-order extract into append-only ClickHouse snapshots, with a verifier that reads the live Tableau dashboard to prove it shows the newest data.
Read the write-up →Asked by Chicago teams.
How do you work with teams in Chicago?+
We are in Bengaluru and move our working day for Illinois, so four hours of every working day overlap with your morning in Central time, stand-up included. Calls, reviews and decisions happen in that window. The rest of the work lives in your tools: messages in Slack, code in GitHub, tickets in Linear. The person on every call is the engineer who writes the code, so a question about the system gets answered by the person who built it.
Do you build for Chicago freight and logistics companies?+
Yes. The work is usually the layer that turns messy events into a shipment you can trust: EDI load tenders, status updates and invoices parsed into one event model, carrier tracking APIs reconciled against it, and proof-of-delivery photos from a field app that works offline. Around that sit the operations screens a brokerage floor uses all day, and the pipelines that turn loads and lanes into pricing and margin reports.
How does your rate compare to hiring in Chicago?+
Built In puts the average base salary for a software engineer in Chicago at about $133,000, before benefits and bonus, and you are competing with trading firms for that person. Our published rate is $35/hour, or $5,400 a month for an embedded engineer, with a $5,000 minimum. There is no recruiting time and no employment overhead, and work starts within days. A fixed-price two-week first piece at $2,800 lets you judge us on output.
Can you build alongside our existing systems rather than replacing them?+
Yes, and in Chicago that is most of the work. We read the old system, write down what it actually does, and build the new piece beside it. Work moves across one workflow at a time, with both running until the last piece lands, so the business keeps running without a cut-over weekend. We did exactly this on FM360, moving a pipeline across module by module rather than in one cut-over.
Do we need a data warehouse yet?+
Maybe not. If your reporting runs comfortably against a Postgres read replica, a warehouse adds a second system to pay for and keep in step. The signs you need one: analytical queries slowing the product, data from several sources that has to be joined, or history you must keep that the application overwrites. We will tell you which side of that line you are on.
Why ClickHouse rather than Snowflake or BigQuery?+
Not always. For steady, high-volume analytical work, ClickHouse is fast and cheap to run, self-hosted or on its managed cloud, and we run it in production. Snowflake and BigQuery ask less of you operationally and suit ad hoc querying across a large team. If nobody on your side wants to operate a database, we will recommend the managed option even when the invoice is larger.
What does a freshness guarantee actually mean?+
A written statement per table or report, such as "never more than an hour behind" or "reflects yesterday's close by 07:00". Each one has a check that measures it and an alert that fires when it is missed. Without it, stale data looks exactly like correct data until somebody makes a decision on it.
Can you work with the tools we already have?+
Usually, yes. If you already run dbt, Airflow, Fivetran or a BI tool your team knows, we build inside it. We suggest replacing a tool only when it is the cause of the problem you hired us for, and we show you the evidence before asking you to fund a migration.
How do you handle personal data in the warehouse?+
We decide what should reach the warehouse at all before building the pipeline. Identifiers can be hashed or tokenised at ingestion, sensitive columns restricted by role, and retention enforced by the pipeline itself. Every field is documented from source to dashboard, so anyone can see where a number came from and who can read it.