通过 Flink CDC 加速 Lakehouse 实时数据摄取.pdf

1、Accelerate Data Ingestionin Real-Time Lakehouse via Flink CDCLeonard Xu Flink PMC Member&Flink CDC Lead,Staff Enigeneeralibaba01WhyReal-Time Lakehouse need Flink CDC03HowTo use Flink CDC02WhatIs Flink CDC04NextCommunity&Future PlanWhy Real-Time Lakehouse need Flink CDC?why01Traditional Lakehouse Arc

2、hitecture LogsDatabaseData Lake(S3/OSS)Queries Queries Queries ODSDWDDWS(Batch)(Batch)Pros Flexibility：Open to engines Cost:Low-CostCons Data Freshness:T+1 hour CDC Support:Limited supportData Ingestion Flexibility：Open to engines Cost:Low-Cost for storage Data Freshness:T+3 minutes CDC Support:Pret

3、ty GoodReal-Time Lakehouse:Fresher data is betterProsLogsDatabaseData Lake(S3/OSS)Queries Queries Queries ODSDWDDWSStreamingStreamingFlink CDCA Streaming Data Ingestion Tool is required!The First Step to build Real-Time LakehouseLogsDatabaseData Lake(S3/OSS)Queries Queries Queries ODSDWDDWSStreaming

4、StreamingFlink CDCSteps to build Real-Time LakehouseData IngestionStreaming IngestionSchame EvolutionFull DB SyncYAML APIFlink CDCData StoragePaimonUpsert/Partial-UpdateChangelog ProducingTime TravelLookup JoinData ComputationFlinkStreaming ComputingBatch ComputingChangelog ProcessingSQL APIUser Que

5、riesStarRocks/SparkOpen to OALP enginesExternal tableMemory optimizationWhat is Flink CDC?what02What is Flink CDC?Flink CDC is an end to end streaming data ingestion tool that implements unifiedsnapshot reading and incremental reading based on the CDC(Change DataCapture)technology of database logs.F

6、link CDCReal-time snapshotFlink CDCSnapshot DataChange LogPaimonIcebergTraditional CDC Ingestion PipelineDataX/SqoopSnapshot SyncDebezium/CanalChangelog SyncMergeMerged TableIncremental TableSnapshot TableData INFRAsData ConsistencyData FreshnessData StackDBFlink CDC Ingestion PipelineCanal/Debezium