Introduction
Open Universities Australia (OUA), a not-for-profit organization founded in 1993, has empowered over 500,000 students through online learning opportunities across 25 Australian universities. Recently, OUA undertook a significant transformation of their data infrastructure, moving away from third-party ETL tools to AWS native services.
The Challenge and Solution
Facing high costs from third-party ETL tools, OUA identified an opportunity to replicate functionality using AWS services while improving code management and visibility. The migration strategy focused on two parallel tracks: building new architecture and migrating existing pipelines.
Guiding Principles
- Simple and modular design patterns with minimal complexity
- Cost-effective resource utilization
- Business continuity through gradual implementation
Key AWS Services Implemented
- Amazon Redshift for centralized data warehousing
- AWS CDK for infrastructure code management
- AWS Step Functions for workflow orchestration
- AWS Glue for data integration
- Amazon AppFlow for SaaS application integration
- AWS Lambda for serverless computing
Implementation Patterns
The solution incorporated four main patterns:
- Pattern 1: Optimized data transformation and loading using Step Functions
- Pattern 2: Data replication with AWS Glue for third-party sources
- Pattern 3: Streamlined data replication using Amazon AppFlow
- Pattern 4: Reverse ETL for external system data export
Impressive Results
The migration was completed in just 5 months, achieving significant cost reductions compared to the previous third-party solution. The transition maintained seamless operation for end users while improving pipeline visibility and maintenance efficiency.
Future Impact
The modernized architecture has transformed OUA’s data operations, enabling faster processing, improved scalability, and enhanced decision-making capabilities. The code-based approach provides greater flexibility and control over data pipelines while maintaining cost-effectiveness.