Real-Time Sales Analytics Pipeline
Sales reporting ran on overnight batches, so the business was always looking at yesterday. I rebuilt it as an event-driven pipeline on AWS that processes sales data as it arrives and lands it in a governed lake, with orchestration that explains itself when something goes wrong.
What was actually broken
Batch processing meant decisions were made on stale numbers, and the monolithic jobs were hard to debug. When a step failed there was no clean retry and no clear signal of what broke, so failures turned into morning fire drills.
How it fits together
Event-driven ingestion
EventBridge and Lambda react to new sales data the moment it lands in S3, kicking off processing instead of waiting for a nightly window.
Governed lake, three zones
A Raw to Staging to Curated layout on S3, governed by Lake Formation, so access is controlled and each zone has a clear job.
Glue transformation
AWS Glue handles the heavy transformation between zones, scaling with the volume instead of a fixed batch box.
Metadata-driven orchestration
Step Functions reads pipeline metadata to wire up dependencies, retries, and alerting, so the flow is configurable rather than hard-coded.
The parts that did the work
- Metadata-driven Step Functions orchestration, so adding a new feed is a metadata entry rather than a rewrite.
- Automatic retries with SNS alerting, so transient failures self-heal and real ones page someone with context.
- A Lake Formation governed lake that keeps access controlled across the Raw, Staging, and Curated zones.
The hard parts, and how I got past them
Failures in long-running batch jobs were opaque and expensive to recover from.
Breaking the flow into Step Functions states with per-step retries and SNS alerts turned silent failures into recoverable, observable events.
Every new data source used to mean new orchestration code.
Driving the workflow from metadata meant new sources and dependencies are described in configuration, not bespoke code.
What changed for the business
What I took awayEvent-driven beats batch the moment freshness matters, but the real win was making orchestration metadata-driven. Once the pipeline could describe itself, extending it stopped being an engineering project each time.
More work
Let's build something that ships.
Hiring for a senior Data/AI role, or need a data platform that actually holds up in production? Let's talk.
or email me directly at muhammaduzairkhan329@gmail.com