Data engineering in Sydney
Month end is where a Sydney financial firm finds out whether its data agrees with itself. The same loans, deposits and policies feed the finance team, the risk team and the board, and in many firms they reach each one through a different extract and a different spreadsheet. When two reports show different numbers for the same book, someone senior spends days working out which one is right.
We build pipelines that land each source once, with one agreed definition of each figure, lineage from every reported number back to the system it came from, and checks that stop a report when a feed is late or incomplete. On FM360 we learned to monitor what the reader sees rather than trusting the job that feeds it, because a pipeline can report success while the numbers are stale. We bring that habit to financial reporting.
Sydney is where Australian money keeps its head office. Commonwealth Bank, Westpac and Macquarie run their groups from the city, the large insurers are here, and around them sits a dense fintech sector in payments, lending and wealth. The software these firms need is rarely a brand-new product. It is the work around the core: payment flows that settle in seconds, onboarding and lending journeys customers actually finish, reports that agree with each other, and the internal tools that keep claims and complaints moving.
Real-time payments have raised the bar. On the New Payments Platform money moves in seconds, and PayTo lets a business take agreed payments straight from a customer's account, so a lender or fintech built around overnight batch files now needs instant confirmations and a ledger that never records the same payment twice. Sydney is also where Atlassian and Canva started, so its buyers have been trained by good software built down the road. A clunky internal tool or a slow sign-up flow is noticed here in a way it might not be elsewhere. Beyond finance, the city has a large property and strata sector, online retailers shipping across a very big country, and a startup scene around Tech Central.
Hiring reflects all of that. Indeed puts the average base salary for a software engineer in Sydney at A$119,602, from 526 reported salaries, and experienced engineers are contested by the banks, the big product companies and every well-funded startup at once. A team that needs a payments engineer or an extra pair of senior hands for a quarter can wait a long time for the right hire. That is where we are useful: a defined piece of work, started within days, with the code in your repositories from the first commit.
Our four hours of daily overlap land in your afternoon, and we move them to suit your team. Your morning stays with your own people; the afternoon is when you talk to the engineer writing your code, in stand-ups, reviews and design decisions. Runbooks and architecture notes are written as we go, so whoever joins your team next inherits work they can read.
We are the right fit when you have a defined piece of work and want senior engineers on it within days, or when the Sydney hire you need is months away and the roadmap keeps moving. Start with a fixed-price two-week piece and judge us on the output. Agencies can put the work under their own name.
Four ways this arrives.
Two definitions of the same metric, computed in two places. We pick one, write it as a tested model in version control, and point every report at it, so the argument moves from whose number is right to what the definition should be.
A job reports success while the report still shows yesterday. We give every table a written freshness guarantee, monitor what the reader actually sees rather than the job that feeds it, and alert a person when the guarantee is missed.
A CRM, an ERP or a work-order tool reachable only through its API. We build the sync with retries, rate limits and normalisation at the boundary, and keep append-only snapshots so last Tuesday is a query rather than a restore.
Analytical queries running against the application database at the worst possible moment. We move them to a columnar store, keep Postgres for transactions, and feed one from the other by change data capture or scheduled extracts, whichever the freshness requirement calls for.
A nightly work-order extract into append-only ClickHouse snapshots, with a verifier that reads the live Tableau dashboard to prove it shows the newest data.
Read the write-up →Asked by Sydney teams.
How do you work with teams in Sydney?+
We are in Bengaluru and share four hours of live overlap with you every day, in your afternoon, moved to suit your team. That is when we join stand-ups, review pull requests and take design calls. We work in your tools, whether that is Slack, GitHub or Linear, and the person on every call is the engineer who writes the code. Between calls, progress is in your tracker and your repositories.
Do you build for Sydney banks, insurers and fintechs?+
Yes, and that is most of what Sydney asks us for. We build payment backends that handle NPP and PayTo without recording a payment twice, sign-up and lending journeys customers finish on their phone, complaint and claims tools that take the reading off skilled staff, and data pipelines that make finance and risk reports agree. We work inside your codebase and your cloud, starting with the one flow that matters most.
How does your rate compare to hiring in Sydney?+
Indeed puts the average base salary for a software engineer in Sydney at around A$120,000. Our published rate is $35/hour with a $5,000 minimum, or $5,400 a month for one embedded engineer. There is no recruiting time and no employment overhead, and work starts within days of the call. The first piece is a fixed-price two-week engagement at $2,800, so you judge us on what ships.
Can you take over an app another team built?+
Yes. It is a common starting point for Sydney products whose agency or founding engineers have moved on. We read the code, run it, and write down how it fits together and where the risks sit. The first fixed-price two-week piece is usually one real fix or feature, so you see the handover working while the architecture notes build up. The code stays in your repositories throughout, as it always would.
Do we need a data warehouse yet?+
Maybe not. If your reporting runs comfortably against a Postgres read replica, a warehouse adds a second system to pay for and keep in step. The signs you need one: analytical queries slowing the product, data from several sources that has to be joined, or history you must keep that the application overwrites. We will tell you which side of that line you are on.
Why ClickHouse rather than Snowflake or BigQuery?+
Not always. For steady, high-volume analytical work, ClickHouse is fast and cheap to run, self-hosted or on its managed cloud, and we run it in production. Snowflake and BigQuery ask less of you operationally and suit ad hoc querying across a large team. If nobody on your side wants to operate a database, we will recommend the managed option even when the invoice is larger.
What does a freshness guarantee actually mean?+
A written statement per table or report, such as "never more than an hour behind" or "reflects yesterday's close by 07:00". Each one has a check that measures it and an alert that fires when it is missed. Without it, stale data looks exactly like correct data until somebody makes a decision on it.
Can you work with the tools we already have?+
Usually, yes. If you already run dbt, Airflow, Fivetran or a BI tool your team knows, we build inside it. We suggest replacing a tool only when it is the cause of the problem you hired us for, and we show you the evidence before asking you to fund a migration.
How do you handle personal data in the warehouse?+
We decide what should reach the warehouse at all before building the pipeline. Identifiers can be hashed or tokenised at ingestion, sensitive columns restricted by role, and retention enforced by the pipeline itself. Every field is documented from source to dashboard, so anyone can see where a number came from and who can read it.