Technology
Data pipelines you can rerun, test and trust
Python pipelines usually start as scripts and grow into something the business depends on. We give them orchestration, quality checks and clear contracts, so a failure is loud, contained and safe to rerun.
- Duration
- 4 to 12 weeks per stage
- Model
- Staged programme
- Output
- Tested pipelines and data contracts
Signs you need this
Each of these shows up in job logs, dashboards or the questions analysts ask.
- Cron jobs on a single server run the pipelines, and when one fails the next one runs on bad data.
- A column renamed in a source system breaks reports days later, and nobody can say which tables were affected.
- Rerunning a failed job creates duplicate rows, so every rerun needs a person and a spreadsheet.
- Two dashboards disagree because the same metric is calculated in several SQL scripts.
- The nightly batch no longer finishes before the working day, and streaming is proposed as the fix.
- Each Python project pins different library versions, and nobody can rebuild last month’s environment.
- A model works in a notebook, but its features are computed differently in production.
Most data failures are quiet
A script that moves data does not fail like a web service. It finishes, reports success and writes the wrong numbers. Without checks on schema, volume, freshness and nulls, the first alarm is a person in a meeting who does not believe a chart.
The usual fixes are a bigger server, a move to streaming or a new tool. None of them changes the real causes: jobs that are not idempotent, transformations scattered across scripts and nobody owning the shape of the data. We start with orchestration, such as Airflow or Dagster, that makes dependencies and retries explicit; contracts between the teams that produce data and the pipelines that read it; and one SQL transformation layer, for example with dbt, tested like code.
The data model and the stores around it, such as partitioning or which store serves which workload, belong to Data Architecture and Migration. This service is about the Python and SQL that move and shape the data, and the services that expose it.
What is included, and what is not
Included
- Orchestration with Airflow, Dagster or your current scheduler, including retries and backfills
- Idempotent loads, so any run can be repeated without duplicates
- Data quality checks on schema, volume, freshness and nulls, routed to the owning team
- Data contracts between source systems and the pipelines that read them
- A tested SQL transformation layer with one definition per metric
- Feature pipelines and FastAPI services that use the same logic for training and serving
- Packaging and dependency hygiene: lock files, pinned images and one way to run tests
Not included
- Building BI dashboards and reports
- Training or tuning machine learning models
- Choosing a warehouse or lakehouse vendor
- Running the pipelines day to day after handover
How the work runs
- Weeks 1 to 2Map the flows. With read-only access to code, schedulers and warehouse metadata, we map every pipeline: sources, outputs, schedule and who reads the result. Recent failures and reruns show where the risk sits.
- Weeks 3 to 6Make runs safe. The riskiest pipelines move under orchestration with idempotent loads, retries and backfills. Quality checks run at each step and stop bad data before it spreads.
- Weeks 5 to 10Agree the contracts. Producers and consumers agree schemas and change rules. Shared metrics move into one tested SQL layer, and duplicate scripts are retired.
- Weeks 8 to 12Serve the data. Where applications or AI features need the data, FastAPI services and feature pipelines are added with typed schemas, tests and one code path for training and serving.
- ThroughoutKeep access narrow. Access is read-only by default. Any write access used during a stage is removed when it ends.
What you get
All outputs belong to you: code, reports, diagrams and their source files, kept in your repositories.
When this is not the right service
- Teams that need dashboards rather than pipelines. We build the flows that reports rely on; the reports themselves are a separate job.
- Model training and data science work. When an AI feature needs architecture around the model, AI / LLM Integration Architecture is the place to start.
- Data that a single well-indexed database handles comfortably. An orchestrator and a warehouse would add cost without benefit.
Frequently asked questions
Airflow or Dagster?
Both work. Airflow is widely known and fits task-based schedules; Dagster models data assets directly, which suits teams that think in tables rather than tasks. If you already run one, we usually keep it and fix how it is used.
Do we need streaming?
Only where the business acts on data within minutes, such as fraud checks or stock levels. Streaming adds cost and new failure modes, so most reporting stays in batch, designed to agree with the streaming numbers.
How do you handle personal data in pipelines?
We design pipelines to keep personal data in few places and to record where it flows, which makes access and deletion requests under rules such as UK GDPR, GDPR or KVKK easier to answer. This is an engineering measure, not legal advice.
Which Python versions and tools do you use?
Supported CPython releases, one packaging tool per repository and a lock file. Where your team already uses a sound tool, such as Poetry or uv, we keep it rather than add another.
What happens after the stage?
Your team runs the pipelines with the runbook, and write access is removed. The contracts and checks keep working without us, because they run in your pipeline.
Start with a short technical conversation
Thirty minutes. You describe your pipelines, and the architect who would run the work tells you where we would start.