数据摄取快慢：如何通过正确的时间处理提高数据可用性和数据质量.pdf

1、Data Ingestion,Fast and SlowDillon BostwickDatabricks20232023 Databricks Inc.All rights reservedConfidential and Proprietary3The Worlds Data Is Real-Time2023 Databricks Inc.All rights reservedConfidential and Proprietary4The World Needs Real-Time Analytics2023 Databricks Inc.All rights reservedConfi

2、dential and ProprietaryAnd Not JustDigital Transformation52023 Databricks Inc.All rights reservedConfidential and ProprietaryConventional“Batch vs.Streaming”Dichotomy Is Limiting62023 Databricks Inc.All rights reservedConfidential and ProprietaryA Data Engineers LifeFrom:Retailer CEORetailer CEOSubj

3、ect:Need an analysis ASAP!Subject:Need an analysis ASAP!To:Dillon Bostwick Great report!Can you update it every dayevery day?every minuteWeekly Sales Forecast2023 Databricks Inc.All rights reservedConfidential and Proprietary8Process Data at Real Time2023 Databricks Inc.All rights reservedConfidenti

4、al and Proprietary9Process Data at RightTime2023 Databricks Inc.All rights reservedConfidential and Proprietary10Think Right-TimeAdjust Data Freshness to Business Need,at Any SpeedCostLatencySecondsMinutesHoursDaysWeeks1.Reduce risk of decisions on stale data2.Reduce risk of inaccurate models(drift)

5、3.Improve agility in transition to real-time4.Improve cost predictability2023 Databricks Inc.All rights reservedConfidential and ProprietaryStep 1:Scoping Right-Time Use Cases11Fraud DetectionRecommendersConcept DriftReal-Time BiddingComposable CDPRegulatory ReportingIntraday ValuationLoan ApprovalI

6、nventory RebalancingSpoilage ReductionOn-Shelf AvailabilityStock tickerTradingPoint of SaleERP(Inventory)OT(digital twin,IoT)Retail/ManufacturingFinancial ServicesMediaHorizontal/DigitalUse CaseData SourcesDSP*CRMClickstreamUser preferencesLTV*Demand Side Platform2023 Databricks Inc.All rights reser