Lakehouse Pipeline Provider: Streaming CDA Parquet Exports to Apache Iceberg

The Technical Challenge: Ingesting Raw CDA Parquet Exports

Modern SaaS applications and enterprise platforms (such as Salesforce, CRM systems, and ERP platforms) provide Cloud Data Access (CDA) capabilities. CDA regularly pushes bulk Parquet file exports directly into cloud object storage (Amazon S3 buckets). While raw Parquet export is superior to legacy CSV files, converting these continuous dumps into reliable analytical assets poses severe engineering challenges:

• The High-Frequency Small File Problem: CDA feeds continuously emit thousands of small Parquet files every hour. Querying unmanaged S3 buckets filled with micro-files degrades query performance in Amazon Athena or Redshift and spikes S3 GET API request costs.
• Unordered Change Records & Missing ACID Guarantees: CDA exports contain inserts, updates, and deletes across partitioned folders. Without an open table format, performing row-level upserts requires scanning and rewriting entire historical datasets.
• Silent Schema Drift: When source application teams add columns or mutate field data types, naive ETL scripts crash, leading to silent pipeline failures and corrupted analytical tables.
• High Provisioned Compute Costs: Maintaining 24/7 dedicated clusters (EMR or EC2 instances) to poll S3 buckets for intermittent CDA file landings generates excessive infrastructure waste.

CDA Parquet to Apache Iceberg Architecture

High-level ingestion pipeline flow: Continuous streaming and micro-batch reading of Cloud Data Access (CDA) Parquet export files into production Apache Iceberg format tables on AWS S3, with unified catalog discovery across AWS Glue Data Catalog and Databricks Unity Catalog:

End-to-End Solution: Serverless AWS Glue CDA to Apache Iceberg Pipeline

As a specialized lakehouse pipeline provider, we engineer resilient, production-hardened data pipelines tailored to your SLA, volume, and budget requirements. Our turnkey solution bridges raw CDA exports and your downstream analytical lakehouse:

Production Pipeline Code & Configuration

Our pipelines are built with clean, modular PySpark code running natively on AWS Glue 4.0+. Below is an example of our streaming micro-batch CDA consumer merging incoming event updates directly into Apache Iceberg:

1# AWS Glue Serverless PySpark Pipeline: CDA Parquet to Apache Iceberg
2from pyspark.sql import SparkSession
3from pyspark.sql.functions import col, current_timestamp
4
5# Initialize Spark Session with AWS Glue Data Catalog & Iceberg extensions
6spark = SparkSession.builder \
7    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
8    .config("spark.sql.catalog.glue_catalog", "org.apache.iceberg.spark.SparkCatalog") \
9    .config("spark.sql.catalog.glue_catalog.warehouse", "s3://<Bucket Name>/iceberg-warehouse/") \
10    .config("spark.sql.catalog.glue_catalog.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \
11    .config("spark.sql.catalog.glue_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \
12    .getOrCreate()
13
14# 1. Read Raw Stream Raw Parquet Data from landing bucket
15cda_landing_path = "s3://<Bucket Name>/raw/salesforce_cda/year=*/month=*/*.parquet"
16
17raw_cda_df = spark.readStream \
18    .format("parquet") \
19    .schema(cda_inferred_schema) \
20    .option("maxFilesPerTrigger", 500) \
21    .load(cda_landing_path)
22
23# 2. Enrich, Deduplicate, and Cast Data Types
24transformed_df = raw_cda_df \
25    .filter(col("id").isNotNull()) \
26    .withColumn("ingestion_timestamp", current_timestamp())
27
28# 3. Micro-Batch Upsert (MERGE INTO) Target Apache Iceberg Table
29def write_iceberg_microbatch(microbatch_df, batch_id):
30    microbatch_df.createOrReplaceTempView("cda_updates")
31    spark.sql("""
32        MERGE INTO glue_catalog.lakehouse_db.cda_customer_events target
33        USING cda_updates source
34        ON target.id = source.id
35        WHEN MATCHED AND source.updated_at > target.updated_at THEN
36            UPDATE SET *
37        WHEN NOT MATCHED THEN
38            INSERT *
39    """)
40
41query = transformed_df.writeStream \
42    .format("iceberg") \
43    .trigger(processingTime="1 minute") \
44    .option("checkpointLocation", "s3://<Bucket Name>/checkpoints/cda_events/") \
45    .foreachBatch(write_iceberg_microbatch) \
46    .start()
47
48query.awaitTermination()

Serverless Cost Optimization Profile

We configure automated worker scaling so you never overpay for fluctuating CDA export volumes:

1# Serverless AWS Glue Job Execution Configuration (Terraform / CLI)
2Type: "G.1X"                  # 4 vCPU, 16 GB memory per DPU
3NumberOfWorkers: 2            # Autoscales between 2 to 10 workers dynamically
4Timeout: 60                   # Caps execution duration for batch schedules
5GlueVersion: "4.0"            # Native Spark 3.3.0 & Iceberg support
6DefaultArguments: {
7  "--datalake-formats": "iceberg",
8  "--enable-auto-scaling": "true",
9  "--job-bookmark-option": "job-bookmark-enable",
10  "--conf": "spark.sql.iceberg.handle-timestamp-without-timezone=true"
11}

Why Choose Our Lakehouse Pipeline Solution Over Generic SaaS Tools?

Enterprise data teams often debate whether to use expensive per-GB SaaS data replication platforms, assemble brittle internal shell scripts, or engage a specialized lakehouse pipeline provider. Here is how our architecture compares:

Instant SQL Discoverability: Athena, Redshift, Spark & Trino

By registering Apache Iceberg tables in the AWS Glue Data Catalog, your raw CDA data is immediately ready for self-service analytics across modern SQL query engines without data movement:

Frequently Asked Questions (FAQ)

Get Started with a Production Lakehouse Pipeline

Eliminate small file bloat, unreliable manual scripts, and expensive per-GB SaaS ingestion costs. Deploy a battle-tested serverless AWS Glue Iceberg pipeline for your Cloud Data Access exports.