Java与大数据架构：3. Sync Clickhouse with MySQL_MongoDB.pdf

1、Sync Clickhouse with MySQL/MongoDB Company: Xiaoxin Tech. Industry: Education Team: Big Data Leader: About 100 billion data this year till now 30 million users We use Clickhouse in our daily tasks Chanllenges Complex Datasource Chanllenges Frequent Updates Chanllenges Possible Solutions 1. Replay bi

2、nlog/oplog CRUD directly Cant update/delete table frequently in Clickhouse Possible Solutions 2. MySQL Engine Not suitable for big tables Not suitable for MongoDB Possible Solutions 3. Reinit whole table every day Possible Solutions 4. CollapsingMergeTree FINAL is slow GROUP BY id HAVING sum(sign)0

3、Need to use GROUP BY in every query Not suitable for multi-column primary key Our Solution: PTS Key Features Only one config file needed for a new Clickhouse table Init and keep syncing data in one app for a table Sync multiple data source to Clickhouse in minutes PTS Provider Transform Sinker Major Provider Must Listen Merge sharding tables SolidKey Typical Provider Config Type: mysql,/ mysql, m