Product
Modern data platform
One platform for structured, streaming and unstructured data, built on the medallion architecture. Bronze, silver and gold, with tagging, ownership and access policies applied at each layer, so everything downstream reads from a single certified set of tables.
Stage 01
Sources - start with an inventory, not a connector
Which systems hold the numbers, who owns them, how often they change, and where definitions differ between them. That map comes before any pipeline is written.
- ERP, finance and HCM systems
- Operational databases and application back ends
- SaaS APIs and third-party feeds
- Files and drops: CSV, Excel, JSON, XML, fixed-width
- Events, telemetry and streaming sources
- Documents and other unstructured content
Stage 02
Ingest - built for the awkward cases
Ingestion is where most of the engineering effort actually goes. Schemas evolve, sources backdate transactions, and jobs need to restart cleanly. We design for all of that from the start rather than meeting it in month four.
- Continuous file ingestion as data arrives in cloud storage
- Change data capture from operational systems, so loads stay incremental
- Schema evolution handled explicitly - new columns land and the run continues
- Idempotent and replayable loads: rerunning never double-counts
- Records needing review are quarantined and stay visible
- Alerting that reaches a person, with the context to act on it
Medallion - bronze
Bronze - the truth as the source told it
Append-only and immutable. No cleaning, no business rules, no opinions. Raw exists so every downstream number traces back to precisely what arrived, and so the whole platform can be rebuilt from nothing if it has to be.
- Full source fidelity, including the malformed rows
- Tagged on landing: source system, load id, ingest timestamp, classification
- Complete history retained - replay any date, any layer, from here
- Open table formats, so corrections are versioned rather than overwritten
- Access restricted to the platform team by policy, not by convention
Medallion - silver
Silver - where the rules get applied
Types enforced, duplicates resolved on real business keys, reference data conformed, and quality rules applied with exceptions routed for review rather than dropped. This is the layer that establishes what your data says.
- Quality expectations, with a review table for anything that does not meet them
- Deduplication on business keys, not whole-row hashes
- Slowly changing dimensions handled properly, with history preserved
- Conformed reference data: one customer list, one chart of accounts, one owner each
- Source-to-target reconciliation counts produced on every run
- Personal and sensitive columns tagged and masked here, before anything downstream reads them
Medallion - gold
Gold - one definition, one set of policies
Star schemas and aggregates shaped for how the business asks questions, with every metric defined once, certified, and published under a policy that says who may read it. This is the only layer most people ever touch.
- Facts and conformed dimensions, modelled for querying not storing
- A metric layer: revenue and margin mean one thing everywhere
- Certified tag and a named owner on every published table, discoverable in the catalog
- Row-level and column-level policies enforced at the table, not rebuilt in each report
- Pre-aggregated tables where dashboards need speed
- The only layer reporting, analytics or AI is permitted to read
Stage 06
Serve - three ways in, one set of numbers
The same governed layer serves all three audiences: executives who want a dashboard, analysts who want SQL and notebooks, and everyone else who would rather just ask a question.
- Dashboards and semantic models built on the curated layer
- SQL and notebooks for analysts, on right-sized compute
- Forecasting, segmentation and anomaly detection on reconciled history
- Natural-language querying that returns the generated SQL alongside the answer
- Every route reads the same certified tables, so numbers cannot diverge
The idea
Bronze, silver, gold
Each layer answers a different question. Keeping them separate is what makes the platform auditable, and what lets any figure be traced end to end years later.
Bronze answers what arrived
Immutable and complete. The question you need when a figure is queried nine months later, and the reason you never have to ask a source system to re-extract.
Silver answers what is true
Cleaned, conformed, deduplicated. Every rule applied here is written down and testable, and every exception stays visible rather than dropped.
Gold answers what it means
Modelled and certified, with definitions the business agreed to before the build - not ones invented inside a report.
Storage and compute stay separate
Query capacity scales with the workload rather than the data volume, so a heavy month does not mean a permanently larger bill.
Governance
Tagging and policies, not good intentions
Governance only works when it is enforced by the platform rather than remembered by people. Every rule below is applied at the table, so it holds no matter which tool the data is read from.
Classification tags
Every table and column tagged for sensitivity as it lands: public, internal, confidential, personal. The tag travels with the data, so a policy written once applies everywhere that column ends up.
Ownership and stewardship
Each dataset carries a named owner and a steward in the catalog, so there is always someone to ask about a definition and someone who approves access to it.
Column masking policies
Personal and financial columns masked by default and unmasked only for roles explicitly granted it. Applied centrally, so a new report cannot accidentally expose what an old one hid.
Row-level policies
Region, entity or business-unit filters bound to the table itself, so one certified dataset can serve teams who must not see each other's rows.
Certification
Gold tables marked certified, with an owner and an agreed definition. Anything uncertified is visibly uncertified, so nobody builds a board pack on a draft.
Lineage and retention
Column-level lineage from bronze through to the report, and retention rules per layer - which is what makes an audit question or a deletion request answerable.
Fit
When this is right, and when it is not
A good fit when
- You have a mix of structured, semi-structured and streaming data
- Machine learning or advanced analytics is on the roadmap
- Data volumes are large or unpredictable
- You want open table formats and no long-term lock-in
- Several teams share the same data and must not see the same rows
Probably not when
- Everything you report on is structured and already in relational systems
- Your team is SQL-first with no appetite for notebooks or Python
- The requirement is finance and operations reporting, nothing more
- A warehouse would do the job at lower cost - we will say so
Tools
Platforms we work on: Databricks, Microsoft Fabric, Snowflake and the major cloud platforms.
Ready to build a platform?
Starting fresh or extending what you already have, tell us where you are and we will suggest the next step.