谷歌：2024大语言模型合成数据的最佳实践和经验教训报告（英文版）（26页）.pdf

1、2024-04-10Best Practices and Lessons Learned onSynthetic Data for Language ModelsRuibo Liu1,Jerry Wei1,Fangyu Liu1,Chenglei Si2,Yanzhe Zhang3,Jinmeng Rao1,Steven Zheng1,DaiyiPeng1,Diyi Yang2,Denny Zhou1and Andrew M.Dai11Google DeepMind,2Stanford University,3Georgia Institute of TechnologyThe success

2、 of AI models relies on the availability of large,diverse,and high-quality datasets,whichcan be challenging to obtain due to data scarcity,privacy concerns,and high costs.Synthetic data hasemerged as a promising solution by generating artificial data that mimics real-world patterns.Thispaper provide

3、s an overview of synthetic data research,discussing its applications,challenges,andfuture directions.We present empirical evidence from prior art to demonstrate its effectiveness andhighlight the importance of ensuring its factuality,fidelity,and unbiasedness.We emphasize the need forresponsible use

4、 of synthetic data to build more powerful,inclusive,and trustworthy language models.1.IntroductionFigure 1|One synthetic image generated by Imagen(Saharia et al.,2022a)v2.0,with a promptincluding the following description:“In a robotics factory,humanoid robots collaborate on an assemblyline to desig

5、n,fabricate,test,and assemble new robots.The new robots they are manufacturing looksimilar to those robotic workers who are creating them.”We also added some style controlling textfrom aesthetic considerations.Corresponding author(s): 2024 Google DeepMind.All rights reservedarXiv:2404.07503v1 cs.CL

6、11 Apr 2024Best Practices and Lessons Learned on Synthetic Data for Language ModelsThe rapid advancement of artificial intelligence(AI)technologies has led to their widespreadadoption across numerous domains,from assistant agents(e.g.,ACT-1,from Adept AI1)and softwaredevelopment(e.g.,Devin,from Cogn