【微软】A Unified Database for Scalar-Vector Data.pdf

1、A Unified Database for Scalar-Vector Data Management张虔熙目录 Approximate Nearest Neighbor Search Query Support-VBase Index Update-SPFreshApproximate Nearest Neighbor SearchPART 1Vectorization:An Important Trend In AI Era 1,3,5,4,.2,8,1,0,.9,2,5,6,.High-dimensional VectorsDeep Learning ModelRaw DataQARe

2、commendationLLM Retrieval PluginVector similarity search empowers semantic-understanding tasksQuery AnsweringVector Similarity SearchANNS for Large-scale Vector SearchVector Search:finding k nearest vectors to a query vectorExact search is extremely expensive and unscalable!Approximate Nearest Neigh

3、bor Search(ANNS)The only viable system solution to large-scale vector searchLow query latency(milliseconds)and high search accuracy(90%+)Indices are invented to achieve ANNSIVFFlatHNSWSPANNDiskANNTopK PrimitiveANNS query primitive:TopKSELECT*FROM items ORDER BY embedding 3,1,2 LIMIT 5;Current“vector

4、 database”is still far from perfectANNS Index(IVFFlat,HNSW)TopK-K tuplesN=5?YesDoneIndex ScanCRUD operationsCurrent solutionsReadLimited Query SupportUpdate/DeleteInefficient Update/DeleteQuery Support-VBasePART 2Limited Query Support SELECT*FROM items WHERE category_id=123 ORDER BY embedding 3,1,2

5、LIMIT 5;Inefficient!Optimal K is hard to predict.But this kind of query demand is very common.SELECT*FROM items WHERE embedding 3,1,2=5?YesDoneANNS IndexTopK-K tuplesIndex ScanTopK primitive is insufficient to support complex vector analytics systematicallyHow Scalar Index WorksSELECT*FROM items WHE

6、RE category_id=123 ORDER BY score LIMIT 5;B+tree index traversal data exhibits monotonicity,so it can use an iteratorinterface to traverse data to decide K during runtime.Filtercategory_id=123N tuplesN=5?YesDoneB+treeIndex ScanIterator-GetNextOSDI 23 VBase:Unifying Online Vector Similarity Search an