Fifteen Systems, No Single Truth
Our client is a fast-growing multinational specialty retailer operating across Australia and New Zealand, with a physical store network and an online operation. Like most retailers of this size, they had grown by adopting best-in-class systems for each function — around fifteen of them in total.
Lightspeed for point of sale. Shopify for e-commerce. NetSuite for ERP and finance. Cin7 for inventory. Deputy for workforce management. Salesforce for CRM, bookings, and customer appointments. Plus specialist tools across learning management, safety and compliance, footfall analytics, web analytics, paid advertising across Google and Meta, and 3CX for the call centre and customer service.
Each system did its job. None of them talked to each other.
Reporting was a weekly exercise in manual extraction and spreadsheet reconciliation. Store performance, marketing effectiveness, stock positioning, staff productivity, customer behaviour — every question that mattered required someone to pull data from three places and hope the numbers added up.
Starting Small: A Two-System Pilot
We did not start with a grand architectural plan. We started with a focused pilot covering two systems: Lightspeed and Deputy. Sales data from the point of sale and staffing data from the workforce management tool, brought together and made visible to the people who needed to act on it.
The impact was immediate. For the first time, information was in the hands of the people running the business in near real-time. Store managers could see how their day was going while there was still time to influence it. Regional managers could spot under-performance and act on it the same week, not two weeks later when the report finally arrived. Information that had previously been visible only to a few people, days after the fact, became visible to everyone who needed it.
Decisions stopped being retrospective. Actions started influencing performance instead of just explaining it.
That pilot is what unlocked the wider investment. It proved the value before anyone had to commit to a large-scale build. And it set the principle that has shaped every part of what came after: get the right information to the right people, fast.
Foundations That Evolved
We did not jump straight from the pilot to a full data warehouse. After the pilot proved its value, we built an expansive BI layer directly over the core applications — Lightspeed, Shopify, NetSuite, Deputy. Using their data, their structures, their constraints. This delivered serious value quickly. Whole parts of the business got onto trusted reporting for the first time without having to wait for a multi-year platform build.
As the business grew, the data complexity grew with it. More systems. More countries. More questions that crossed multiple sources. More volume than the source systems could comfortably serve directly. That is when we made the move to a full Databricks platform with a Medallion architecture: Bronze for raw ingestion, Silver for cleaned and harmonised data, Gold for business-ready datasets.
The Gold layer is where the real value sits now. Every metric, every dashboard, every report draws from it. Definitions are agreed once and applied consistently across the business. When a regional manager and a finance director both look at store performance, they are working from the same numbers, calculated the same way.
It has been a real journey. A focused pilot. An expansive BI layer over core applications. Proper data platform engineering when complexity demanded one. Each step earned its place.
The point worth pulling out is that you do not have to build the final form on day one. Start with what delivers value now. Evolve the foundations as the business needs grow. The wrong move is to over-engineer too early — building a full data platform when a focused BI layer would have done the job, or skipping the data platform when the business has outgrown direct integrations.tructured summaries plus searchable semantic indexes, turned out to be one of the most important design decisions in the whole build.
BI for Retail: A Business Running on a Single Platform
Today, every part of the business makes decisions based on the BI built on this platform.
Over a dozen purpose-built Power BI Apps cover every function: live store dashboards used by store managers throughout the day, sales performance and people reporting used across the store network, an HQ App covering forecasting, live sales, capacity, inventory, stocktakes, employee retention and demand. A stripped-back Executive App with the Chairman’s Weekly Report and top-level forecasting. Finance Apps for budgets and resource allocation. Marketing environments for Google and Meta analytics with full funnel analysis, campaign performance, and the link between digital activity and in-store sales.
The difference between dashboards people open because they have to and dashboards people open because they want to is design. Each App is built for the audience that uses it, not built once and pushed to everyone.
Row-level security ensures the right people see the right data. Refresh schedules are tuned to each App’s use case. The store network teams get near real-time updates. Finance teams get end-of-day reconciled views. Marketing teams get daily campaign performance. Each audience gets what they need at the cadence they need it.
This is not “we built a few dashboards.” This is a properly mature enterprise business intelligence for retail deployment, running every operational, financial, and commercial decision in the business.
The Right Pattern for Each Source
None of this works if the data is not flowing reliably. Different sources have different shapes, different access patterns, different freshness requirements. Treating them all the same produces fragile pipelines that break when anything changes.
The platform uses three ingestion tracks, each matched to the kind of source it is handling.
Native Databricks pipelines for cloud-based sources with strong connectors: Google Analytics and Google Ads through BigQuery, Salesforce Marketing Cloud via Data360, NetSuite via Suite Connect, inbound feeds in Azure Storage. Managed connectors do the heavy lifting reliably and cheaply.
Custom ETL pipelines built and maintained by us for operational SQL databases that need more careful handling: Lightspeed, NetSuite core tables, Shopify, Deputy, Cin7. Orchestrated through Azure Data Factory. Full control over what gets pulled, how it gets transformed, how errors are handled. These pipelines also serve as the source for the app-to-app integrations.
A bridging layer for edge cases like the PostgreSQL database powering the phone system, which feeds into the SQL layer via a separate pathway.
This three-track approach is what makes the platform sustainable. New sources are added using the pattern that fits them, not shoehorned into a single one-size-fits-all approach.r request. In cost and latency terms, that is small. In flexibility terms, it is enormous.
Beyond Reporting: An Operational Integration Layer
Properly built data integration solutions can do more than report. It can run the business.
The same platform underpins a network of app-to-app integrations between the core operational systems. Every retail sale at the point of sale flows automatically into the ERP. Every product in the ERP catalogue flows out to the point of sale. Every online order fulfilment flows into the ERP. Every store-to-store stock transfer flows in as a transfer order. All of it happens automatically, every day, across both Australia and New Zealand.
These integrations carry real business logic, not just data feeds:
- Bidirectional with the right source of truth for each thing. The point of sale is authoritative for sales transactions. The ERP is authoritative for the product catalogue, inventory, and finance. The integration respects these boundaries.
- Multi-region, multi-currency. Australian sales post in AUD to the AU ledger. New Zealand sales post in NZD to the NZ ledger. Local timezones. Live exchange rates for cross-region supplier pricing.
- Complex business rules baked in. Split tenders, returns inside normal sales, gift cards, cash rounding, tax mismatch reconciliation at the cent level. All handled automatically.
- Audit logging and error isolation operations teams can trust. Every transaction logged. Failures don’t block other transactions. Fifteen transactions in parallel.
- Idempotent design. Re-running an integration cannot create duplicate financial records. Finance teams never have to clean up after a retry.
This is the difference between a “data warehouse for reporting” and a genuine operational platform. The same foundation that powers the dashboards also runs the integrations that keep the business operating end to end.
At full operating volume, this layer processes thousands of sales and inventory transactions every day across the store network and online channel. No manual handling. No end-of-day batch reconciliation. The dashboards reflect what actually happened in the business, not what someone manually pulled together overnight.
The platform also serves live operational workloads in real time. A Bookings API exposed on top of the same data layer powers customer-facing booking applications, returning live capacity and availability on demand. The same data feeds the dashboards leadership and store managers use to track demand and conversion. One platform, one source of truth, serving both operational systems and reporting from the same foundation.
Three Years In: The Platform Keeps Earning
The original build went live some time ago. The platform has not stood still. New stores added. New countries opened. New systems integrated. New use cases delivered. The dashboards have evolved with the business.
The clearest illustration of how this compounds is the current Marketing Analysis programme. A brand-new initiative looking at marketing journeys, campaign performance across Google, Meta, and Salesforce Marketing Cloud, customer acquisition patterns, and the link between marketing spend and revenue across the entire business.
Critically, this project is not a from-scratch build. It is being constructed on top of the existing platform. Common dimensions for date, location, and product already exist in the Gold layer. Customer data from every system already links through agreed identifiers. The marketing-specific layer being added slots in beside the existing model rather than replacing anything.
That means the project moves faster, costs less, and produces output consistent with the rest of the business reporting. The platform earns more every year it runs.
The Broader Lesson
A business data solution is a long-term asset. The architecture choices made at the start determine how much value comes out over time.
The lessons we would pull out for any business in a similar position:
- Start small. Prove value early. A focused pilot on a small number of systems gets information to the people who need it fast, and earns the investment for what comes next.
- Evolve the foundations with the business. Early on, a focused BI layer over the core applications can deliver real value. As complexity grows, a full data platform — Bronze, Silver, Gold — earns its place. It is a journey, not a single build.
- Match the ingestion pattern to the source. Cloud sources, operational databases, and edge cases each need different handling.
- Build dashboards with the people who will use them. Generic templates get abandoned. Dashboards designed around the actual questions of each function get used every day.
- Treat the platform as an operational asset, not just a reporting one. The same foundation can run the integrations that keep the business operating.
- Plan for what comes next, not just what you need now. Every new project should build on the platform, not start from scratch.
None of this is unique to retail. The shape will be different for a SaaS business, a professional services firm, or a manufacturer. The principles are the same: start small, foundations first, the right pattern for each source, build with the users, design for the long term. The organisations that get this right end up with a platform that keeps serving them years after the initial investment, and that keeps absorbing new use cases without being rebuilt.
