The Challenge of Financial Data Integration
Amazon Finance Automation (FinAuto) faced a significant challenge: integrating financial transactional data from multiple systems, each with their own data models, into a unified operational data store. The goal was to support financial operations applications with millisecond latency while processing hundreds of millions of daily transactions.
Key Requirements for the Financial Data Store
The solution needed to address several critical requirements:
- Scale rapidly to handle massive transaction volumes
- Deliver millisecond-level API response times
- Support flexible schemas to accommodate various sub-ledgers
- Enable comprehensive searching across financial entities
- Provide aggregation capabilities across multiple dimensions
- Store and manage account relationships for consolidated customer views
A Multi-Database Approach
To address these diverse requirements, Amazon’s Finance team implemented a solution combining multiple AWS purpose-built databases:
Amazon DynamoDB as the Primary Store
DynamoDB served as the primary data store, providing low-latency key-value access to financial records. Its flexible schema accommodated different attributes from various financial systems, while on-demand capacity mode handled unpredictable workloads across regions.
The team implemented:
- UUID v5 identifiers to ensure uniqueness across systems
- Global Secondary Indexes (GSIs) for efficient queries on non-primary attributes
- Local Secondary Indexes (LSIs) to optimize sorting operations
- Native DynamoDB exports to Amazon S3 for reconciliation processes
Amazon OpenSearch Service for Advanced Querying
To enable search-as-you-type functionality and complex aggregations, the team used:
- DynamoDB Streams to capture data changes
- Lambda functions to process these events
- SQS queues to decouple processing
- Edge N-gram tokenizers for efficient prefix searching
This approach allowed financial analysts to quickly identify accounts with high aging receivables and perform efficient search operations.
Amazon Neptune for Relationship Management
Account relationships presented a unique challenge: customers often have multiple accounts but require unified billing. Amazon Neptune’s graph database capabilities provided:
- Natural modeling of account hierarchies and relationships
- Efficient traversal queries with a single request
- Flexibility to extend relationship types over time
- Sub-250ms response times for relationship queries
Ensuring Operational Excellence
For a system handling critical financial data, reliability was paramount. The team implemented:
- Dead-letter queues for failed message processing
- CloudWatch monitoring for API errors and latency
- Custom metrics for business-specific monitoring
- Regular data reconciliation with source systems
Key Benefits of the Solution
This purpose-built approach delivered significant advantages:
- Scalable performance handling hundreds of millions of daily transactions
- Consistent low-latency responses for user applications
- Flexible data models accommodating diverse financial systems
- Powerful search and aggregation capabilities
- Comprehensive relationship management without custom graph solutions
By leveraging purpose-built AWS databases and serverless computing, Amazon’s Finance team created a solution that satisfied their unique requirements while maintaining high performance and data quality.
For more information about how Amazon Finance Automation built an operational data store with AWS purpose-built databases, visit the AWS Database blog
Leave a Reply