Case study · Industry
Building a bronze-silver-gold lakehouse for an international law firm
A medallion-architecture lakehouse in Azure Synapse, giving the data science team the right level of granularity for whatever they're analysing.
Azure Synapse Analytics
Medallion Architecture
Azure Key Vault
Azure DevOps
GitHub Actions
3
Data layers, bronze/silver/gold, one governed lakehouse
On time
Every pipeline ready for the live environment
Tracked
Every schema change, as source tables evolved
The client
The customer was an international law firm looking to build a lakehouse in Azure Synapse under a medallion architecture: a data design pattern with bronze, silver and gold layers, each progressively improving the structure and quality of the data as it flows through.
The purpose was to let the data science team move freely between granular data in the silver layer and aggregated data in the gold layer, depending on what a given piece of analysis needed.
The challenges
- •Synapse limitations: the customer had existing Data Factory integration runtimes already built, but Synapse doesn’t support sharing those between instances, so they couldn’t simply be reused.
- •Communication: multiple layers of stakeholders and instances needed to work in tandem, particularly around networking and access control, requiring close, regular work with the client’s own IT team.
- •Architecture changes: as the lakehouse was built, many source tables were migrated or replaced, bringing in schema changes between old and new versions that had to be tracked and documented throughout.
What we delivered
- ✓A full Azure Synapse deployment as the core of the lakehouse.
- ✓Bronze, silver and gold data layers, built to a medallion architecture.
- ✓Knowledge transfer to the client’s own team.
The outcome
- ✓The medallion architecture, though challenging to set up, proved ideal for surfacing data at different levels of granularity for different audiences.
- ✓Unexpected data issues came up along the way, as they do on a build like this, but the team stayed on top of them quickly enough that milestones never slipped.
- ✓Development stayed on schedule, with every required pipeline in place and ready for the live environment.
The founder’s take
“A medallion architecture is easy to draw and hard to run. The real value is in the discipline of tracking every schema change as source systems evolve underneath you, because once that's solid the data science team stops asking IT for data and just gets on with using it.”
Thomas Madden
Founder & CEO, IO Analytics