Background and Challenges
Vanguard faced significant challenges with their legacy mainframe system that limited their ability to deliver modern, personalized customer experiences. Their centralized database architecture created performance bottlenecks and made it difficult to scale services independently for millions of personal and institutional investors.
The existing mainframe system, built on a centralized Db2 database, enabled cross-domain data access but introduced several architectural challenges:
- Resource Contention: Processes from one domain negatively impacted other domains due to shared compute resources
- Lack of Domain Isolation: Changes in one bounded context created unintended ripple effects across other domains
- Scalability Constraints: Centralized architecture created bottlenecks as load increased
- High Coupling: Tight integration made it challenging to modify individual components
- Limited Fault Tolerance: Issues in one domain cascaded across the entire system
The ORDS Solution with Amazon Redshift
To address these challenges, Vanguard implemented an Operational Read-only Data Store (ORDS) using Amazon Redshift. This solution leverages Amazon Redshift’s compute and storage separation to create multi-cluster architectures with separate endpoints for each domain.
Key benefits include:
- Resource Isolation: Each domain receives dedicated Amazon Redshift compute resources
- Independent Scaling: Domains can scale compute resources based on specific needs
- Controlled Data Sharing: Amazon Redshift’s data sharing enables secure cross-domain data access without tight coupling
Implementation Architecture
The ORDS architecture implements Amazon Redshift data sharing through two key patterns:
Data Ingestion: Vanguard utilized Amazon Redshift’s zero-ETL functionality alongside AWS Database Migration Service (AWS DMS) for Amazon Aurora PostgreSQL databases and Amazon Kinesis for Amazon DynamoDB data streaming.
Data Sharing: Once data is ingested, they created views for consumers and established data shares with read-only access to consumer Amazon Redshift data warehouses for batch processing.
Real-World Application
A practical example is their brokerage account statement generation process, which requires integrating data from multiple bounded contexts including Account details, Client Profile information, and Transaction History. This cross-bounded context batch process accesses hundreds of tables and processes large volumes of data monthly.
Business Outcomes
The transition to ORDS with Amazon Redshift has delivered significant improvements:
- Enhanced client experience through improved data management and accessibility
- Empowered autonomous teams with a resilient batch architecture
- Efficient cross-domain data access with high-quality data consistency
- Near-real-time access to large data volumes with high performance and cost-effectiveness
This modernization effort successfully transformed Vanguard’s batch processing capabilities while maintaining seamless data access across different business domains, enabling them to deliver superior customer experiences in the financial services industry.
Visit the original AWS blog post for more detailed information