Data engineering in London
Month end is where a London firm finds out whether its data can be trusted. The same transactions feed finance, risk and the board, and in most firms they reach each one through a different spreadsheet, so the numbers agree only after someone has spent a week reconciling them. The next question follows quickly: which dashboards and services depend on which pipelines, and what happens when one stops.
We build pipelines that land data once, with lineage you can follow from the report back to the source system, and checks that fail loudly when a feed is late or wrong. On FM360 we learned to monitor what the reader sees rather than the job that feeds it, because a refresh can report success while the dashboard still shows yesterday. We bring that habit to every reporting pipeline.
Financial services drive more of London's software demand than any other sector. Payments companies, challenger banks, wealth platforms and brokers need ledgers that reconcile, APIs that stay correct when a partner retries, and apps customers trust with their money. Most of them reached the market fast, which was the right call at the time, and now run systems built for launch rather than for the size they have become. The work is making those systems dependable without stopping the business that runs on them.
Insurance is its own case. The specialty insurers, managing agents and brokers around Lloyd's still run much of their work on ageing policy administration platforms and spreadsheets that travel between firms by email. Lloyd's has moved away from Blueprint Two, the market-wide programme that was meant to digitise placement and claims, so each firm is modernising its own systems one integration at a time. The City's law firms and consultancies sit alongside them, holding decades of documents and few ways to search them by meaning.
Media, retail and the public sector bring different problems. Publishers run subscriptions, paywalls and content systems that have to hold up on a busy news day. Direct-to-consumer brands outgrow their first shop as they start selling into Europe and the US. Suppliers to councils and NHS bodies build services that have to work for every resident, including people using a keyboard or a screen reader. Hiring is the common constraint. ITJobsWatch puts the median advertised salary for a software engineer in London at £98,750 over the six months to September 2026, and banks, fintechs and the large technology firms compete for the same people.
Working with us from London is straightforward. Four hours of every working day overlap with yours, set across your morning, so stand-up, code review and the decisions that follow happen live. You talk to the engineer who writes the code, in your own Slack, GitHub and Linear. The rest of the day moves forward in pull requests you can read at your desk, with runbooks and architecture notes written as the work goes.
We are the right fit when you have a defined piece of work and need senior engineers on it this month rather than next quarter. Start with a fixed-price two-week piece and judge us on what ships. We add capacity while a London hire is still in progress, and we work white-label under your name for agencies.
Four ways this arrives.
Two definitions of the same metric, computed in two places. We pick one, write it as a tested model in version control, and point every report at it, so the argument moves from whose number is right to what the definition should be.
A job reports success while the report still shows yesterday. We give every table a written freshness guarantee, monitor what the reader actually sees rather than the job that feeds it, and alert a person when the guarantee is missed.
A CRM, an ERP or a work-order tool reachable only through its API. We build the sync with retries, rate limits and normalisation at the boundary, and keep append-only snapshots so last Tuesday is a query rather than a restore.
Analytical queries running against the application database at the worst possible moment. We move them to a columnar store, keep Postgres for transactions, and feed one from the other by change data capture or scheduled extracts, whichever the freshness requirement calls for.
A nightly work-order extract into append-only ClickHouse snapshots, with a verifier that reads the live Tableau dashboard to prove it shows the newest data.
Read the write-up →Asked by London teams.
How do you work with teams in London?+
Four hours of every working day overlap with yours, set across your London morning, so stand-up and code review happen live. We join your calls and work in your tools: Slack, GitHub and Linear. You talk directly to the engineer who writes the code. Outside the overlap, the work carries on in pull requests you can read at your desk, and runbooks and architecture notes are written as we go.
Do you build for London fintechs and insurers?+
Yes. For payments companies, challenger banks and wealth platforms we build ledgers that reconcile, webhook handling that stays correct when a provider retries, and React Native apps with biometric sign-in and step-up checks for risky payments. For insurers, managing agents and brokers around Lloyd's we replace spreadsheet workflows with internal tools: bordereaux intake, referral queues and approvals, connected to the policy systems you already run.
How does your rate compare to hiring in London?+
ITJobsWatch puts the median advertised salary for a software engineer in London at roughly £99,000. Our published rate is $35/hour, with a $5,000 minimum engagement. There is no recruiting time and no employment overhead, and we can start within days of the first call. Most clients begin with a fixed-price two-week piece at $2,800, so you judge us on working code before committing to more.
Can you work white-label for a London agency?+
Yes. London agencies bring us in when a client project needs more senior backend, mobile or infrastructure work than the team has free. We work under your name, in your repositories and your client's tools, and you stay the face of the project. The NDA comes before specifics, the code belongs to the client from the first commit, and runbooks and architecture notes are written as we go.
Do we need a data warehouse yet?+
Maybe not. If your reporting runs comfortably against a Postgres read replica, a warehouse adds a second system to pay for and keep in step. The signs you need one: analytical queries slowing the product, data from several sources that has to be joined, or history you must keep that the application overwrites. We will tell you which side of that line you are on.
Why ClickHouse rather than Snowflake or BigQuery?+
Not always. For steady, high-volume analytical work, ClickHouse is fast and cheap to run, self-hosted or on its managed cloud, and we run it in production. Snowflake and BigQuery ask less of you operationally and suit ad hoc querying across a large team. If nobody on your side wants to operate a database, we will recommend the managed option even when the invoice is larger.
What does a freshness guarantee actually mean?+
A written statement per table or report, such as "never more than an hour behind" or "reflects yesterday's close by 07:00". Each one has a check that measures it and an alert that fires when it is missed. Without it, stale data looks exactly like correct data until somebody makes a decision on it.
Can you work with the tools we already have?+
Usually, yes. If you already run dbt, Airflow, Fivetran or a BI tool your team knows, we build inside it. We suggest replacing a tool only when it is the cause of the problem you hired us for, and we show you the evidence before asking you to fund a migration.
How do you handle personal data in the warehouse?+
We decide what should reach the warehouse at all before building the pipeline. Identifiers can be hashed or tokenised at ingestion, sensitive columns restricted by role, and retention enforced by the pipeline itself. Every field is documented from source to dashboard, so anyone can see where a number came from and who can read it.