锟斤拷锟斤拷位锟矫ｏ拷锟斤拷页 > 锟斤拷锟斤拷锟教筹拷 > 锟教筹拷 > llama.cpp锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷小锟斤拷锟叫凤拷锟斤拷业锟侥革拷锟斤拷锟斤拷CPU/GPU锟斤拷锟斤拷锟斤拷模锟斤拷锟斤拷锟斤拷锟斤拷锟�

llama.cpp锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷小锟斤拷锟叫凤拷锟斤拷业锟侥革拷锟斤拷锟斤拷CPU/GPU锟斤拷锟斤拷锟斤拷模锟斤拷锟斤拷锟斤拷锟斤拷锟�

锟斤拷源锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷|锟斤拷时锟戒：2024-10-06 09:55:16 |锟斤拷锟侥讹拷锟斤拷112锟斤拷|锟斤拷锟斤拷签锟斤拷 a 小一锟斤拷 C 锟斤拷锟斤拷锟斤拷锟斤拷 |锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷锟斤拷锟斤拷戏

llama.cpp锟斤拷一锟斤拷锟斤拷锟斤拷锟杰碉拷CPU/GPU锟斤拷锟斤拷锟斤拷模锟斤拷锟斤拷锟斤拷锟斤拷埽锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷鸭锟斤拷璞革拷锟斤拷缘锟借备锟斤拷锟斤拷锟斤拷锟竭匡拷锟斤拷通锟斤拷锟斤拷锟竭斤拷锟斤拷锟洁开源锟斤拷锟斤拷锟斤拷模锟斤拷转锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷gguf锟斤拷式锟斤拷锟侥硷拷锟斤拷然锟斤拷通锟斤拷llama.cpp实锟街憋拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟揭的碉拷锟叫ｏ拷锟斤拷冉锟斤拷锟斤拷锟斤拷锟侥ｏ拷锟斤拷锟截凤拷锟斤拷锟斤拷锟斤拷小锟斤拷锟叫凤拷锟斤拷业使锟斤拷llama.cpp锟斤拷锟斤拷锟斤拷唯一锟侥诧拷品锟斤拷锟�

锟斤拷小锟斤拷锟叫凤拷锟斤拷业锟斤拷锟斤拷隙锟斤拷锟角э拷锟�+锟斤拷硬锟斤拷投锟诫，锟斤拷小锟斤拷锟叫凤拷锟斤拷业只锟斤拷支锟斤拷锟斤拷锟斤拷硬锟斤拷投锟诫，锟斤拷锟斤拷也缺锟斤拷专业锟斤拷锟叫凤拷锟斤拷员锟斤拷

锟斤拷品锟斤拷胤锟斤拷锟斤拷锟斤拷锟侥匡拷锟揭拷弑锟斤拷诖锟街憋拷锟斤拷锟斤拷锟截碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟铰伙拷锟斤拷要私锟叫伙拷锟斤拷锟斤拷

锟斤拷锟斤拷锟叫诧拷锟劫斤拷锟杰碉拷锟斤拷锟铰ｏ拷B站锟斤拷锟斤拷锟斤拷锟斤拷一些锟秸费课程★拷锟斤拷锟角版本锟斤拷锟较多，锟斤拷锟斤拷锟窖撅拷没锟叫参匡拷锟斤拷值锟斤拷锟斤拷锟侥诧拷锟斤拷b3669锟芥本锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷2024锟斤拷9锟铰ｏ拷锟轿匡拷锟斤拷锟诫：examples/main.cpp锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷(Georgi Gerganov)没锟斤拷锟结供锟斤拷细锟侥接匡拷锟侥碉拷锟斤拷examples锟侥达拷锟斤拷锟斤拷锟斤拷也确实锟斤拷锟竭ｏ拷锟斤拷锟窖帮拷锟斤拷弑冉隙锟斤拷汀锟斤拷锟斤拷锟街硷拷诮锟斤拷锟斤拷锟斤拷使锟斤拷llama.cpp锟斤拷锟斤拷锟斤拷锟斤拷锟酵斤拷锟斤拷锟截点函锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷员锟斤拷锟脚ｏ拷锟斤拷锟诫功锟杰伙拷锟叫达拷锟叫撅拷锟斤拷

llama.cpp：一个适用于中小型研发企业的高性能CPU/GPU大语言模型推理框架

一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

1. 锟斤拷锟斤拷锟斤拷锟斤拷

锟皆筹拷锟斤拷锟侥斤拷锟斤拷锟斤拷锟斤拷为锟斤拷锟斤拷锟斤拷锟斤拷锟脚匡拷锟皆分筹拷5锟斤拷锟接癸拷锟斤拷模锟介。

2. 锟斤拷锟斤拷锟斤拷锟�

锟斤拷色锟斤拷roles锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷模锟斤拷通锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟街斤拷色锟斤拷系统锟斤拷system锟斤拷锟斤拷锟矫伙拷锟斤拷user锟斤拷锟斤拷锟斤拷锟街ｏ拷assistant锟斤拷锟斤拷锟斤拷锟斤拷锟街斤拷色锟斤拷锟斤拷锟斤拷锟斤拷模锟斤拷统一指锟斤拷锟斤拷锟斤拷锟角伙拷锟斤拷目前锟斤拷锟叫匡拷源锟侥达拷模锟酵讹拷锟斤拷锟斤拷锟斤拷锟斤拷锟街斤拷色锟侥斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟节达拷模锟酵革拷锟矫碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤境锟斤拷锟斤拷锟斤拷锟斤拷锟絪ystem锟斤拷示系统锟斤拷示锟绞ｏ拷锟斤拷锟斤拷锟斤拷锟角筹拷说锟斤拷prompt锟斤拷锟斤拷锟斤拷锟叫诧拷锟劫课程斤拷写系统锟斤拷示锟斤拷锟斤拷锟斤拷为锟斤拷示锟绞癸拷锟教ｏ拷锟斤拷煞锟叫斤拷锟铰的斤拷锟叫凤拷锟洁，锟斤拷实锟斤拷刹锟斤拷亍锟斤拷锟斤拷业锟绞癸拷镁锟斤拷榭达拷锟揭伙拷锟斤拷玫锟较低筹拷锟绞撅拷剩锟絧rompt锟斤拷应锟竭憋拷锟斤拷锟斤拷要锟姐即锟缴ｏ拷锟斤拷锟斤拷锟斤拷确锟斤拷锟斤拷式锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷虻ァ锟斤拷锟斤拷锟斤拷锟饺凤拷锟斤拷锟较低筹拷锟绞撅拷锟斤拷芯锟斤拷锟斤拷锟揭癸拷锟侥ｏ拷锟斤拷锟斤拷傻拇锟斤拷铮拷锟斤拷嘶锟剿碉拷锟斤拷恰锟斤拷锟斤拷锟斤拷锟剿碉拷锟斤拷锟斤拷锟斤拷锟斤拷式锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷使锟斤拷markdown锟斤拷锟斤拷json指锟斤拷一些锟斤拷要锟斤拷锟筋。锟斤拷锟斤拷锟斤拷锟揭拷么锟侥ｏ拷桶锟斤拷锟侥筹拷锟斤拷潭锟斤拷锟斤拷探锟斤拷蟹锟斤拷锟斤拷锟斤拷锟斤拷锟绞癸拷锟絤arkdown锟侥憋拷锟斤拷锓拷锟斤拷锟斤拷锟斤拷锟斤拷要锟斤拷锟斤拷模锟酵讹拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷薪峁癸拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟绞癸拷锟絡son锟斤法锟斤拷锟斤拷锟斤拷虻ゼ锟斤拷锟揭拷么锟侥ｏ拷痛锟斤拷锟斤拷呒锟教拷锟斤拷踊锟斤拷锟斤拷锟斤拷锟教拷锟斤拷锟斤拷锟今。达拷模锟酵碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷全锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟解，锟斤拷锟斤拷锟斤拷锟竭憋拷锟较革拷锟斤拷锟斤拷锟较的筹拷锟斤拷执锟斤拷锟竭硷拷锟斤拷锟斤拷学锟斤拷锟斤拷锟竭硷拷锟斤拷锟斤拷锟斤拷锟轿裁达拷锟斤拷锟斤拷锟斤拷蚀锟侥ｏ拷停锟�1.11锟斤拷1.8谁锟斤拷锟绞憋拷锟斤拷锟斤拷锟揭伙拷锟斤拷锟斤拷锟斤拷母锟斤拷锟斤拷悖拷锟斤拷锟斤拷锟斤拷锟斤拷锟揭伙拷锟斤拷锟斤拷时锟津，斤拷锟斤拷要锟饺斤拷小锟斤拷锟斤拷锟街ｏ拷锟斤拷为11锟斤拷锟斤拷8锟斤拷锟斤拷锟�1.11锟斤拷锟斤拷1.8锟斤拷锟斤拷么锟斤拷锟斤拷锟斤拷锟斤拷锟绞碉拷锟饺肥碉拷锟揭恍╋拷锟斤拷锟斤拷锟斤拷锟斤拷锟接碉拷锟斤拷锟斤拷锟斤拷要锟斤拷锟斤拷锟斤拷么锟届？锟揭的斤拷锟斤拷锟斤拷锟斤拷牵锟斤拷锟斤拷锟津交伙拷锟酵讹拷态锟叫伙拷锟斤拷锟斤拷锟侥★拷锟斤拷锟斤拷系统锟斤拷色锟斤拷锟解，锟矫伙拷一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

3. 锟斤拷锟斤拷峁�

llama.cpp锟侥筹拷锟斤拷峁癸拷冉锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷锟斤拷锟絣lama锟斤拷ggmll锟斤拷ggml通锟斤拷llama锟斤拷锟叫碉拷锟矫ｏ拷锟斤拷锟斤拷通锟斤拷锟斤拷锟斤拷直锟斤拷使锟矫★拷锟斤拷llama锟叫讹拷锟斤拷锟剿筹拷锟矫的结构锟斤拷秃锟斤拷锟斤拷锟絚ommon锟角讹拷llama锟叫猴拷锟斤拷锟斤拷锟杰碉拷锟劫次凤拷装锟斤拷锟斤拷时锟斤拷锟金到凤拷锟斤拷锟斤拷玫锟侥匡拷摹锟斤拷锟斤拷前姹撅拷锟斤拷锟斤拷希锟絚ommon锟叫的猴拷锟斤拷锟戒化锟较快，锟斤拷玫姆锟斤拷锟斤拷强锟斤拷锟斤拷锟斤拷毯锟街憋拷拥锟斤拷锟絣lama.h锟叫的猴拷锟斤拷锟斤拷

4. 源锟斤拷锟斤拷锟�

锟斤拷锟斤拷锟斤拷锟斤拷examples/main/main.cpp锟斤拷为锟斤拷锟斤拷锟斤拷锟截碉拷锟斤拷锟斤拷锟�

(1) 锟斤拷始锟斤拷

全锟街诧拷锟斤拷锟斤拷锟斤拷锟斤拷峁癸拷锟斤拷锟揭拷锟斤拷锟斤拷锟斤拷锟斤拷没锟斤拷锟斤拷锟酵猴拷锟斤拷锟斤拷锟斤拷锟斤拷始锟斤拷模锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥★拷

系统锟斤拷始锟斤拷锟斤拷锟斤拷锟斤拷

系统锟斤拷源锟酵放猴拷锟斤拷锟斤拷

锟斤拷锟斤拷模锟酵猴拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷

锟斤拷锟斤拷锟斤拷锟斤拷common.h锟叫★拷锟斤拷锟斤拷锟斤拷锟揭拷锟侥ｏ拷秃锟斤拷锟斤拷锟斤拷姆挚锟斤拷锟斤拷锟斤拷锟斤拷锟绞癸拷锟絣lama.h锟叫碉拷锟斤拷锟斤拷锟斤拷锟皆猴拷锟斤拷锟斤拷

锟斤拷锟斤拷ggml锟斤拷锟竭程池ｏ拷锟斤拷锟斤拷锟斤拷炭锟斤拷芎锟侥ｏ拷图锟斤拷锟斤拷泄兀锟斤拷锟斤拷锟斤拷锟矫伙拷卸锟斤拷锟斤拷锟斤拷锟较革拷锟斤拷停锟�

锟斤拷锟斤拷锟斤拷锟揭伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷llama.cpp锟斤拷实锟斤拷锟斤拷锟斤拷锟斤拷锟侥存储锟斤拷锟饺★拷锟斤拷锟斤拷锟斤拷锟斤拷谢锟斤拷锟角帮拷锟斤拷遣锟斤拷芑锟侥ｏ拷停锟斤拷医锟斤拷状锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷没锟斤拷锟斤拷锟斤拷prompt锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷裕锟斤拷锟斤拷锟绞碉拷锟斤拷锟斤拷锟斤拷牡亩锟教拷谢锟斤拷锟�

锟斤拷锟剿ｏ拷锟叫癸拷系统锟斤拷始锟斤拷模锟斤拷墓锟斤拷锟斤拷丫锟斤拷锟缴★拷

(2) 锟矫伙拷锟斤拷锟斤拷

为锟剿斤拷锟斤拷锟矫伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷源锟诫集锟叫讹拷锟斤拷锟剿硷拷锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟街达拷模锟酵达拷喽济伙拷锟斤拷锟饺凤拷锟斤拷锟斤拷encodec

(3) 锟斤拷锟斤拷预锟斤拷

锟斤拷锟斤拷预锟解部锟街的猴拷锟侥达拷锟斤拷锟斤拷锟铰ｏ拷锟揭斤拷锟斤拷锟斤拷锟斤拷注锟斤拷锟斤拷session锟斤拷锟竭硷拷删锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟街碉拷锟竭硷拷锟斤拷

锟竭硷拷锟斤拷锟截碉拷锟角ｏ拷锟斤拷锟饺ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷某锟斤拷瘸锟斤拷蓿锟斤拷岫拷锟斤拷锟斤拷锟斤拷锟斤拷帧锟绞碉拷士锟斤拷锟斤拷锌锟斤拷钥锟斤拷锟斤拷毓锟斤拷锟斤拷锟斤拷锟街碉拷锟竭硷拷锟斤拷锟斤拷危锟矫匡拷锟斤拷锟斤拷锟斤拷锟斤拷锟揭伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷疲锟絥_batch锟斤拷锟斤拷锟斤拷锟斤拷要锟斤拷为锟剿碉拷一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟教拷啵低筹拷锟斤拷锟斤拷诔锟绞憋拷锟斤拷锟斤拷锟接︼拷锟斤拷锟斤拷每锟斤拷锟斤拷锟斤拷锟斤拷桑锟絜mbd锟斤拷锟结被锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷珊锟斤拷锟斤拷息锟结保锟斤拷锟斤拷ctx锟叫★拷

(4) 锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟街碉拷源锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷郑锟�

锟斤拷锟斤拷要锟斤拷注锟斤拷2锟斤拷锟街ｏ拷锟斤拷一锟轿碉拷锟竭硷拷锟角斤拷锟矫伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫ｏ拷锟斤拷锟斤拷锟矫伙拷锟斤拷锟斤拷锟诫不锟斤拷要锟斤拷锟斤拷锟斤拷锟斤拷锟街伙拷锟揭拷锟斤拷锟絣lama_sampling_accept锟斤拷锟斤拷锟斤拷锟斤拷1锟斤拷锟斤拷只锟叫碉拷锟矫伙拷锟斤拷锟诫都锟斤拷锟斤拷院锟脚伙拷锟斤拷耄匡拷尾锟斤拷锟揭伙拷锟絫oken锟斤拷写锟斤拷embd锟斤拷锟斤拷锟斤拷锟斤拷毯头锟斤拷锟皆わ拷饨伙拷锟斤拷锟叫ｏ拷直锟斤拷锟斤拷锟斤拷eos锟斤拷

chat_add_and_format锟斤拷锟斤拷只锟斤拷锟斤拷锟斤拷锟叫斤拷锟斤拷锟斤拷锟教硷拷录锟斤拷char_msgs锟叫ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷没锟斤拷影锟届。锟斤拷锟揭碉拷锟斤拷没锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤处锟斤拷锟斤拷

锟斤拷锟斤拷锟截硷拷锟斤拷锟斤拷

通锟斤拷gpt_params锟斤拷始锟斤拷llama_model_params

锟斤拷锟斤拷锟斤拷模锟斤拷指锟斤拷

锟斤拷锟斤拷ggml锟竭程池猴拷锟斤拷锟斤拷锟竭程筹拷

通锟斤拷gpt_params锟斤拷始锟斤拷llama_context_params

锟斤拷锟斤拷锟斤拷锟斤拷蟹执什锟阶拷锟斤拷锟絫oken

锟斤拷取锟斤拷锟斤拷token

锟斤拷锟斤拷锟斤拷锟斤拷token锟斤拷锟斤拷锟斤拷预锟斤拷

执锟叫诧拷锟斤拷锟酵斤拷锟秸诧拷锟斤拷

锟斤拷token转锟斤拷锟斤拷然锟斤拷锟斤拷

锟叫讹拷锟斤拷锟斤拷锟角凤拷锟斤拷锟斤拷锟阶拷猓拷锟斤拷token锟斤拷锟杰猴拷llama_token_eos锟斤拷取锟侥诧拷一锟铰★拷锟斤拷锟揭伙拷锟揭拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫讹拷

锟斤拷锟斤拷锟杰斤拷

锟斤拷锟斤拷旨锟节斤拷锟斤拷llama.cpp锟侥伙拷锟斤拷锟矫凤拷锟斤拷锟斤拷锟斤拷Georgi Gerganov锟斤拷锟铰较快，锟斤拷缺锟斤拷锟侥碉拷锟斤拷锟斤拷丝锟斤拷锟斤拷锟叫╋拷锟斤拷筒锟斤拷锟阶既凤拷锟斤拷锟斤拷锟斤拷叶钥锟杰和憋拷锟侥革拷锟斤拷趣锟斤拷锟皆革拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟桔★拷

小锟斤拷锟狡硷拷锟侥讹拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥斤拷为锟斤拷锟斤拷锟斤拷息锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷同锟斤拷锟睫观碉拷锟街な碉拷锟斤拷锟斤拷锟斤拷锟�

a 1.0: 锟斤拷锟酵ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷营状态锟斤拷锟斤拷式锟斤拷营锟斤拷锟斤拷锟斤拷锟皆ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷; 锟斤拷戏锟斤拷锟斤拷

锟斤拷戏锟斤拷锟�

锟斤拷戏锟斤拷频

锟斤拷戏锟斤拷锟斤拷

锟斤拷戏锟筋动; 锟斤拷alittletotheleft锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟筋备锟杰伙拷迎锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷戏锟斤拷锟斤拷业锟斤拷锟斤拷锟斤拷嵌锟斤拷粘锟斤拷锟斤拷锟斤拷械母锟斤拷锟斤拷锟斤拷锟斤拷锟�