Our semi-structured data pipeline ETL job will allow you to process and transform data from sources like MongoDB, and AWS DynamoDB into your centralized data lake. This enables you to gain deeper insights from your data and improve your decision-making using AI/ML, Data Insights or Data Analytics.

MongoDB Ingestion: We use the MongoDB Connector for Apache Kafka to read data from MongoDB (CDC) and push it to a Kafka topic. From there, we can consume the data via streaming or batch jobs based on business requirements.

For batch processing, we utilize GCP Cloud Dataflow, AWS Glue, or Azure Data Factory, creating configurable jobs for each MongoDB collection to write into the Data Lake in open table formats like Apache Iceberg, Apache Hudi, or Delta Lake. For real-time needs, Spark Streaming jobs can consume directly from Kafka.

DynamoDB Ingestion: A similar concept applies to DynamoDB, which has a Kinesis connector to capture changes (CDC) and push them to a Kinesis stream. We can then consume this data via batch or streaming jobs.

For batch jobs, we use cloud-native ETL tools (Dataflow, Glue, ADF) to read from Kinesis and write to the Data Lake in your preferred open table format. Spark Streaming can also be used for real-time ingestion.Specially for the DDB, we can replace the Kinesis with Kafka as well, We need to write the AWS Lambda function to read the data from DynamoDB Stream and push it to Kafka topic. Example of incremental batch job for DynamoDB to Data Lake using Kafka:Let's see our high level design flow for the semi-structured data sources to data lake.