基于 Flink SQL + Paimon 构建流式湖仓新方.pdf

1、 Flink SQL+Paimon Contents0102 Apache Paimon03Flink+Paimon 01 Hive Lakehouse操作方便 ACID Time Travel Schema Evolution查询更快 Fast Plan Data Skipping时效性好 Upsert 更新 流批一体HUDIIcebergData Lake(OSS/S3)LakehouseStreamingFlinkLakehouseFlinkLake StoreApache Iceberg Append Format V2 Flink Apache Hudi Flink Spark De

2、lta Lake Append MergeInto Flink Apache Paimon Flink Table Store Flink +SparkPosition Delete File Equality Delete File()Data File(Parquet/ORC)+LSM(RocksDB,Clickhouse,Doris,StarRocks)Apache Paimon02Paimon+LSMNew Future on CloudCDC Paimon Flink Streaming LSM Flink Table Store 2022.12023.12023.32023.9 H

3、udi 3 Flink Table Store 0.3 Apache Paimon 0.4 Apache CDC Append Paimon 0.5 Contributor100+Star1400+Fork570+*数据来自GitHub Paimon项目统计Paimon VS Hudi5 MORCOW ()03006009001200CP 5 CP 2 CP 1 CP 30 Paimon COWHudi COW12 X12 XHudi MOR ()0750150022503000BucketBucketPaimonHudi MOR4.2 X4 X2 X1 COW Hudi MOR Compac

4、tion Paimon Compaction14 X*来自阿里云测试数据Paimon Convergence of IT Infrastructure,Online Presence of Core Technologies Flink Data and Intelligence Capabilities of Business Applications1Paimon Hudi 3 7 3 Paimon300GB40GB32Paimon CDC CDC*来自阿里云测试数据Flink+Paimon 03 Data Lake(OSS/OSS-HDFS)流式更新入湖流读增量数据批读批写Hive CD

5、C 链路长，组件多，不稳定产出时延：天+合并延时全量增量割裂，存储浪费DataX/Sqoop全量同步(按需)Canal增量同步定时回流（天）定时合并(天)增量表 分区 T全量表 分区 T-1全量表 分区 T增量表 分区 T-1增量表 分区 T+1全量表 分区 T+1定时合并(天)Flink+Paimon CDC 简单稳定，一键同步数据分钟级实时可见全增量一体，存储复用Flink CDC全增量一体同步Paimon 主键表(不分区)Tag T-1Tag TTag T+1Time TravelConsumer S1New Snapshot GeneratingS2S3S4Snapshot Expir

6、ationDELETEStreaming ReadDELETEPaimon Consumer Failover Consumer 没读完，Snapshot 不会被清理无状态重启作业，消费进度依然在Caused by:java.io.FileNotFoundExceptionChangelog(Jason,1)Table(name,count,PRIMARY KEY(name)(Jason,2)(Jason,1)(Jason,2)Retract(Jason,1)Sumcount