Data engineering in San Francisco
An AI product generates a new kind of data exhaust: prompts, retrieved context, model outputs, user feedback, latency and cost, for every request. San Francisco teams usually log it somewhere and never look again, then need it urgently when a customer asks why an answer was wrong or finance asks why the bill doubled.
We build the pipelines that make that data usable. Request logs flow into a warehouse, often ClickHouse, which we also run for FM360, where they can be joined with accounts, usage and billing. Each record carries its customer and model version from ingestion, so a wrong answer can be traced back to the prompt and context that produced it. From there, evaluation sets, cost reports per customer and quality dashboards are queries rather than projects.
San Francisco's software market is, for now, largely an AI market. The city is home to the best-known model companies and to a much larger layer of venture-backed startups building products on top of their models, alongside the SaaS, fintech and developer-tool companies that were here before. Most buyers are young companies with funding, a deadline set by their next raise, and more product ideas than engineers.
What they need is rarely the model. It is everything around it: queues for requests that take most of a minute, streaming that survives a dropped connection, fallbacks when a provider is rate-limiting, a record of what each request cost, and evaluation sets that tell you whether last night's prompt change made things better or worse. Then, often within months, the first enterprise customer arrives, and its IT team asks for single sign-on, automatic provisioning, roles their own admin can manage and an activity log they can export. A backend built for a demo meeting that list is where many pilots stall. We build the product and the infrastructure that make a model useful to paying customers, and the enterprise layer that turns a pilot into a contract.
Built In puts the average base salary for a software engineer in San Francisco at around $181,000, and the model companies compete for the same people with equity most startups cannot match. For a seed or Series A company, every senior hire becomes a search measured in months, and the roadmap waits while it runs. The work that piles up in the meantime is usually well defined: a provider integration, an admin console, a move to usage-based billing, a mobile companion app. That is the work we pick up in days and deliver in pieces you can judge on their own.
We work from Bengaluru with four hours of live overlap every working day, placed across your morning in San Francisco. Stand-ups, design calls and code review happen in that window, with the engineer who writes the code. Everything decided outside it goes into your repository, your tracker and the architecture notes we write as we go, so you start each day knowing what moved.
We fit best when the work is defined and the roadmap will not wait for a hire: the product and infrastructure around a model, the enterprise features a first large customer asks for, or the months before your next engineer starts. Most teams begin with a fixed-price two-week piece. Agencies can bring us in white-label, under their own name.
Four ways this arrives.
Two definitions of the same metric, computed in two places. We pick one, write it as a tested model in version control, and point every report at it, so the argument moves from whose number is right to what the definition should be.
A job reports success while the report still shows yesterday. We give every table a written freshness guarantee, monitor what the reader actually sees rather than the job that feeds it, and alert a person when the guarantee is missed.
A CRM, an ERP or a work-order tool reachable only through its API. We build the sync with retries, rate limits and normalisation at the boundary, and keep append-only snapshots so last Tuesday is a query rather than a restore.
Analytical queries running against the application database at the worst possible moment. We move them to a columnar store, keep Postgres for transactions, and feed one from the other by change data capture or scheduled extracts, whichever the freshness requirement calls for.
A nightly work-order extract into append-only ClickHouse snapshots, with a verifier that reads the live Tableau dashboard to prove it shows the newest data.
Read the write-up →Asked by San Francisco teams.
How do you work with teams in San Francisco?+
We are in Bengaluru and overlap with you for four hours every working day, placed across your San Francisco morning. Stand-ups, design calls and code review happen live in that window, with the engineer who writes the code. We work inside your tools: Slack for conversation, GitHub for code and review, Linear for the plan. Anything decided outside the window is written down there, so nothing depends on memory.
Do you build for AI startups?+
Yes. Most of that work sits around the model rather than inside it: retrieval over your own data, evaluation sets that run on every prompt change, streaming interfaces, cost tracking per customer, fallbacks between providers, and logging that explains an answer after the fact. Then comes the enterprise layer your first large customer asks for: single sign-on, provisioning, admin roles and activity logs, built so the pilot can become a contract.
How does your rate compare to hiring in San Francisco?+
Built In puts the average base salary for a software engineer in San Francisco at about $181,000, before equity and benefits. Our published rate is $35 an hour, with a $5,000 minimum. There is no recruiting search and no employment overhead, and we can start within days. Most teams begin with a fixed-price two-week piece at $2,800, so you judge us on working code before committing to more.
Can you make our prototype ready for real customers?+
Yes, and it is a common place to start here. A prototype built fast for a demo usually needs the same things: model calls moved into queued, retried jobs, costs recorded per request, tests around the parts that change most, and deploys from CI. We read the code, send a written plan within a week, then take the most urgent piece as a fixed-price two-week job. The repository is yours throughout.
Do we need a data warehouse yet?+
Maybe not. If your reporting runs comfortably against a Postgres read replica, a warehouse adds a second system to pay for and keep in step. The signs you need one: analytical queries slowing the product, data from several sources that has to be joined, or history you must keep that the application overwrites. We will tell you which side of that line you are on.
Why ClickHouse rather than Snowflake or BigQuery?+
Not always. For steady, high-volume analytical work, ClickHouse is fast and cheap to run, self-hosted or on its managed cloud, and we run it in production. Snowflake and BigQuery ask less of you operationally and suit ad hoc querying across a large team. If nobody on your side wants to operate a database, we will recommend the managed option even when the invoice is larger.
What does a freshness guarantee actually mean?+
A written statement per table or report, such as "never more than an hour behind" or "reflects yesterday's close by 07:00". Each one has a check that measures it and an alert that fires when it is missed. Without it, stale data looks exactly like correct data until somebody makes a decision on it.
Can you work with the tools we already have?+
Usually, yes. If you already run dbt, Airflow, Fivetran or a BI tool your team knows, we build inside it. We suggest replacing a tool only when it is the cause of the problem you hired us for, and we show you the evidence before asking you to fund a migration.
How do you handle personal data in the warehouse?+
We decide what should reach the warehouse at all before building the pipeline. Identifiers can be hashed or tokenised at ingestion, sensitive columns restricted by role, and retention enforced by the pipeline itself. Every field is documented from source to dashboard, so anyone can see where a number came from and who can read it.