All Case Studies
TechnologyLed by Endrit Basha · Fortune 500 insurer · 2024–present

Moving 30M+ Messages a Day Without Losing a Record

Insurance-grade reliability for data that has to survive an audit.

Sound familiar? A member's dues history disappears, and you find out from the member.

30M+
messages processed daily
33
enterprise data sources
40%
projected storage costs cut
$150K/yr
vendor cost eliminated
The Challenge

A Fortune 500 insurer's data pipelines feed everything downstream, from analytics and reporting to regulatory filings. The data arrives from dozens of enterprise systems at wildly variable volume, and the failure mode is silent. A message drops, a source stalls, and nobody notices until the numbers downstream are wrong. New York's financial regulator, NYDFS, adds a further requirement of provable policy-based data retention. Keeping data too long is a compliance failure, and deleting the wrong data is worse. The engineering standard is easy to state and hard to meet: ingest everything, lose nothing, and prove both.

The Approach
  1. High-throughput distributed ingestion microservice architected and scaled to stream from enterprise message queues into CosmosDB and Snowflake
  2. 33 enterprise data sources onboarded, reliably processing 30M+ messages per day
  3. Adaptive batching, concurrency controls, and backpressure handling engineered to absorb variable load at low latency
  4. Data observability platform and health engine built with lineage-aware checks, providing real-time visibility into failures before they distort downstream analytics
  5. Proactive failure detection and impact analysis that reduced incident resolution time across critical pipelines
  6. Fault-tolerant data retention platform (Go, Docker, workflow orchestration) automating large-scale policy-based deletion to NYDFS compliance standards
  7. Legacy translation system replaced with a custom-built Go service, eliminating a $150K annual vendor dependency
The Outcome

The ingestion platform onboarded 33 enterprise data sources and reliably processes 30M+ messages per day into CosmosDB and Snowflake. On the compliance side, the retention platform automated policy-based deletion to NYDFS standards, reducing audit risk and cutting projected storage costs by 40%. Replacing a legacy translation system with a custom Go service eliminated a $150K-per-year vendor contract. Endrit was also selected as a peer mentor for new engineers.

AWS Certified Cloud PractitionerTestOut CyberDefense ProPeer Mentor for New Engineers
Why this matters for your association

Your member database, dues history, and financial records deserve the same reliability and security discipline an insurer applies to policy data. When associations migrate between AMS platforms, records get lost, dues histories get corrupted, and often nobody notices until a member complains. The observability and reliability practices described here are what prevent that. And when a vendor charges you for something a purpose-built tool does better, we can tell you when building beats renting, because Endrit has made that call at enterprise scale in his role at a Fortune 500 insurer.

Talk to us about protecting member records

Want this for your organization?

We're accepting Founding Partners now. Services are complimentary during our launch period, so you can see the work before you commit a dollar.