Category: Data Engineering
-
Meta’s Agentic Solution for Data Warehouse Access Management
Meta transforms data warehouse access with multi-agent AI system featuring specialized data-user and data-owner agents. The solution addresses growing complexity in data access patterns through context-aware permissions, granular query control, and rule-based risk management while maintaining security guardrails.
-
Modernizing Batch Processing with Amazon Redshift ORDS Architecture
Discover how Vanguard modernized their legacy mainframe system using Amazon Redshift as an Operational Read-only Data Store (ORDS). Learn about their transition to cloud-native, domain-driven architecture while preserving critical batch processing capabilities and enabling cross-domain analytics with improved performance and scalability.
-
Real-Time Baggage Analytics Using AWS Kinesis Data Streams
IBM and AWS developed a framework to modernize airline baggage analytics using Amazon Kinesis Data Streams, DynamoDB, Apache Flink, and QuickSight. This serverless architecture delivers real-time insights, cost savings, enhanced scalability, and improved operational efficiency for airlines.
-
UDA: Netflix’s Unified Data Architecture for Seamless Data Integration
Netflix introduces UDA (Unified Data Architecture), a knowledge graph-based solution that addresses data integration challenges across their expanding business. UDA enables consistent modeling of business concepts, semantic data integration, and automated schema generation across systems like GraphQL and Avro.
-
Capture Data Lineage from dbt, Apache Airflow, and Apache Spark with Amazon SageMaker
Amazon SageMaker now offers enhanced data lineage capabilities compatible with OpenLineage, allowing users to track data flow from tools like dbt, Apache Airflow, and Apache Spark. This integration creates transparency, builds trust, and centralizes governance of data assets in a single place.
-
Amazon PackScan: Revolutionizing Real-Time Sort Center Analytics with AWS Services
Discover how Amazon transformed its logistics operations with PackScan, an AWS-powered platform that reduced data latency from 1 hour to under 1 minute. This real-time analytics solution processes 500,000 scan events per second across 80 sort centers, resulting in 25% increased throughput and 12% reduction in labor hours.
-
Using Amazon Neptune for Real-time Anomaly Detection in Gaming Transactions
Discover how Zupee leveraged Amazon Neptune’s graph database to detect real-time anomalies in gaming wallet transactions. Learn how they overcame relational database limitations to build an integrity system that processes over 1 million daily transactions, identifies suspicious patterns, and ensures incentives reach legitimate users.
-
How Flutter UKI Optimized Data Pipelines with Amazon MWAA
Discover how Flutter UKI transformed their data pipelines by migrating from EC2-based Airflow to Amazon MWAA, managing 5,500 DAGs and 60,000 daily runs with improved stability and reduced operational overhead.
-
How Meta Manages Data Understanding at Scale: A Privacy-First Approach
Meta has developed an innovative approach to data understanding at scale through their Privacy Aware Infrastructure, using a five-step process that includes schematization, metadata prediction, annotation, asset inventory, and continuous maintenance. This system enables Meta to manage millions of data assets while protecting user privacy and driving product innovation.