Data engineering in Boston
In a Boston lab, the data platform is often a shared drive. Plate readers, sequencers and imaging systems each write files in their own format to their own folder, the electronic lab notebook holds the context, and joining the two is a script a computational scientist runs when someone asks. The answer to "which compounds have we tested against this target" depends on who is asked and which folders they remember.
We build pipelines that pick up instrument output as it lands, keep the raw file untouched, parse it into tables with the sample, the experiment and the instrument attached, and link it to the notebook entry that explains it. Postgres holds the records; ClickHouse takes the high-volume measurements when the volume calls for it. Every table carries a freshness statement and a check. The lineage from raw file to reported value is kept, so any number in a report can be traced back to the instrument run that produced it.
Boston's software buyers mostly work in science, medicine, education and money: biotech and pharmaceutical companies in Cambridge and the Seaport, the hospital systems, digital health start-ups, the universities and the companies spun out of them, and the asset managers and insurers downtown. In most of them, software supports something else, an experiment, a patient or a portfolio. The people who commission it are often scientists and clinicians, many of whom write some code themselves and know exactly what they need the data to do. What they want from an engineer is not a new idea. It is the idea they already have, made reliable.
The work follows the science. Biotech labs need instrument output picked up as it lands and linked to the notebook entry that explains it, and internal tools that stay fast on thousands of assay results. Digital health companies need a backend that can read from a hospital's records system, and patient apps that save offline and keep the time each entry was made. Edtech companies selling to universities need platforms that know a teaching assistant from a department administrator. Lab suppliers need commerce that runs on purchase orders and negotiated prices, and the investment firms downtown need internal tools to replace the workbooks that run their operations.
Hiring has its own shape. Built In puts the average software engineer base salary in Boston at about $137,000, and the city's engineers are pulled between big tech offices, well-funded biotech and university spin-outs. The universities keep research talent in good supply. What teams more often lack is someone who has taken a system from a notebook or a prototype to something that runs unattended, is monitored, and survives its author going back to the lab. That gap, between a result that works once and a system that keeps working, is where we are most useful.
We are in Bengaluru and move our working day for Boston. Four hours of every working day overlap with your morning in Eastern time, stand-up included, so questions, reviews and decisions happen live with the engineer who writes the code. Runbooks and architecture notes are written as we go, so the system keeps running when the people who commissioned it go back to the lab.
We are the right fit for the gap between a result that works once and a system that keeps working: a pipeline, a hospital integration, a patient app, an internal tool. Senior capacity starts within days, the first piece is a fixed-price two-week engagement, and it can carry you from a funding round or grant to your first engineering hire.
Four ways this arrives.
Two definitions of the same metric, computed in two places. We pick one, write it as a tested model in version control, and point every report at it, so the argument moves from whose number is right to what the definition should be.
A job reports success while the report still shows yesterday. We give every table a written freshness guarantee, monitor what the reader actually sees rather than the job that feeds it, and alert a person when the guarantee is missed.
A CRM, an ERP or a work-order tool reachable only through its API. We build the sync with retries, rate limits and normalisation at the boundary, and keep append-only snapshots so last Tuesday is a query rather than a restore.
Analytical queries running against the application database at the worst possible moment. We move them to a columnar store, keep Postgres for transactions, and feed one from the other by change data capture or scheduled extracts, whichever the freshness requirement calls for.
A nightly work-order extract into append-only ClickHouse snapshots, with a verifier that reads the live Tableau dashboard to prove it shows the newest data.
Read the write-up →Asked by Boston teams.
How do you work with teams in Boston?+
We are in Bengaluru and move our working day for Boston, so four hours of every working day overlap with your morning in Eastern time, stand-up included. We join your Slack, push to your GitHub and track the work in your Linear, so progress shows up where your team already looks. The person on every call is the engineer who writes the code, whether you are a scientist, a clinician or a CTO.
Do you build for Boston's biotech and life sciences companies?+
Yes. The work is usually the software around the science: pipelines that pick up plate reader, sequencer and imaging output as it lands and link it to the notebook entry, internal tools that stay fast on thousands of assay results, and search over protocols and SOPs that always cites the current version. We work alongside your computational scientists, turning what already works in their notebooks into systems that run unattended.
How does your rate compare to hiring in Boston?+
Built In puts the average software engineer base salary in Boston at about $137,000, before bonus and benefits. Our published rate is $35/hour, or $5,400 a month for an embedded engineer, with a $5,000 minimum. There is no recruiting time and no employment overhead, and work starts within days of the call. The first piece is a fixed-price two-week engagement at $2,800, so you judge us on what it produces.
What do the first two weeks look like?+
You share the context: the code, the data and the problem. Within two days you have a 30-minute call with the engineer who would do the work, and within a week a written plan. Then comes a fixed-price two-week piece at $2,800, such as turning one notebook or prototype into something that runs on its own. At the end you have working code in your repository, notes on how it runs, and a clear view of what comes next.
Do we need a data warehouse yet?+
Maybe not. If your reporting runs comfortably against a Postgres read replica, a warehouse adds a second system to pay for and keep in step. The signs you need one: analytical queries slowing the product, data from several sources that has to be joined, or history you must keep that the application overwrites. We will tell you which side of that line you are on.
Why ClickHouse rather than Snowflake or BigQuery?+
Not always. For steady, high-volume analytical work, ClickHouse is fast and cheap to run, self-hosted or on its managed cloud, and we run it in production. Snowflake and BigQuery ask less of you operationally and suit ad hoc querying across a large team. If nobody on your side wants to operate a database, we will recommend the managed option even when the invoice is larger.
What does a freshness guarantee actually mean?+
A written statement per table or report, such as "never more than an hour behind" or "reflects yesterday's close by 07:00". Each one has a check that measures it and an alert that fires when it is missed. Without it, stale data looks exactly like correct data until somebody makes a decision on it.
Can you work with the tools we already have?+
Usually, yes. If you already run dbt, Airflow, Fivetran or a BI tool your team knows, we build inside it. We suggest replacing a tool only when it is the cause of the problem you hired us for, and we show you the evidence before asking you to fund a migration.
How do you handle personal data in the warehouse?+
We decide what should reach the warehouse at all before building the pipeline. Identifiers can be hashed or tokenised at ingestion, sensitive columns restricted by role, and retention enforced by the pipeline itself. Every field is documented from source to dashboard, so anyone can see where a number came from and who can read it.