Celeborn社区支持Flink最新进展.pdf

1、Celeborn社区支持Flink最新进展熊佳树阿里云-数据湖构建与分析Outline架构及协议设计基本架构概念统一与设计01核心功能Shuffle生命周期管理网络/读写02性能与稳定性读写性能优化容错与流控设计03进展与未来展望开发进展未来规划0401-基本架构FlinkClientImplDriver ManagerReadClientHa MasterWorkerWorkerWorkerWriteClientCelebornFlinkread/writeslotsregister|split|reserve|release|commitShuffle概念设计FlinkDataSetId(

2、16)PartitionNum(16)ProducerId(16)SparkShuffleIdMapIdAttemptIdCelebornShuffleId(int)MapId(int)AttemptId(int)AppIdJobIDAppIdMapPartition&Reduce PartitionReduce PartitionPush Shuffle+Partition 数据聚合Map Partition受限Flink场景及目前Shuffle接口必须实现部分场景具备优势Map#1Map#2Map#0Partition#1Partition#2Partition#0Sub(1.N)Sub(

3、1.M)Sub(1.T)Map Partition（Mapper）Result PartitionPartition#1Partition#2Reducer#0Map#1Map#2Map#0Partition#1Partition#2Partition#0Sub（M）Reducer Partition（Mapper）Result PartitionPartition#1Partition#2Reducer#0Flink ShuffleMapper#1Partition#1Mapper#2Partition#2Reducer#1Mapper#1Partition#1Mapper#1Partiti

4、on#1Mapper#2Partition#2Reducer#3Reducer#4Mapper#1Partition#1Mapper#2Partition#2Reducer#2Reducer#1Reducer#1Reducer#2Reducer#2协议支持-Result PartitionMap#0_1Map#0_2Map#0_0Partition#(1,_)Partition#(2,_)Partition#(0,_)Map#0_1Map#0_2Map#0_0Map Partition（模式）Reduce Partition模式Celeborn PartitionMap Partition区分

5、相同Map不同attempt的结果:PartitionId(Int)=(8 AttemptId)+(24 MapId)Reduce Partition无需区分Partition:PartitionId=ReducerID02-基本功能资源分配与管理Register Shuffle(Slot allocator)ReplicaPartition Spit/Revive/DestroyShuffle Result GroupShuffle ExpireApplication Expire网络传输控制流/数据流数据读写Flink Client/Map Partition reader/writer/

6、CommitCommit 插件Client 插件Reader 插件Writer 插件MapPartitionReducePartition03-关键技术性能支持客户端压缩&小包合并（LZ4/ZSTD）支持IO Scheduling支持Broadcast优化稳定性流控（Credit-based）内存管理Channel复用容错支持客户端压缩&小包合并（LZ4/ZSTD）减少网络IO减少文件SizeMap(Producer)CompressCelebornWriterReader/IOschedulingMerge/Sp