锟斤拷锟斤拷位锟矫ｏ拷锟斤拷页 > 锟斤拷锟斤拷锟教筹拷 > 锟教筹拷 > LLM 锟斤拷模锟斤拷学习锟斤拷知锟截伙拷系锟斤拷(十锟斤拷)锟斤拷VLLM锟斤拷锟杰凤拷跃锟斤拷锟斤拷实锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟劫碉拷锟斤拷效锟斤拷锟斤拷锟饺拷锟轿伙拷呕锟絒锟斤拷锟斤拷锟斤拷锟捷ｏ拷XInference/FastChat锟饺匡拷锟絔

LLM 锟斤拷模锟斤拷学习锟斤拷知锟截伙拷系锟斤拷(十锟斤拷)锟斤拷VLLM锟斤拷锟杰凤拷跃锟斤拷锟斤拷实锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟劫碉拷锟斤拷效锟斤拷锟斤拷锟饺拷锟轿伙拷呕锟絒锟斤拷锟斤拷锟斤拷锟捷ｏ拷XInference/FastChat锟饺匡拷锟絔

锟斤拷源锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷|锟斤拷时锟戒：2024-06-01 08:46:01 |锟斤拷锟侥讹拷锟斤拷129锟斤拷|锟斤拷锟斤拷签锟斤拷系 T 全十锟斤拷锟斤拷跃 INF v S C in 锟斤拷 |锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷锟斤拷锟斤拷戏

LLM 锟斤拷模锟斤拷学习锟斤拷知锟截伙拷系锟斤拷(十锟斤拷)锟斤拷VLLM锟斤拷锟杰凤拷跃锟斤拷锟斤拷实锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟劫碉拷锟斤拷效锟斤拷锟斤拷锟饺拷锟轿伙拷呕锟絒锟斤拷锟斤拷锟斤拷锟捷ｏ拷XInference/FastChat锟饺匡拷锟絔

训锟斤拷锟斤拷锟侥ｏ拷突锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷卟锟斤拷锟斤拷锟斤拷锟斤拷锟绞癸拷锟侥ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥癸拷锟教ｏ拷锟斤拷锟斤拷锟角斤拷模锟酵凤拷锟斤拷锟斤拷锟姐定锟斤拷锟叫的伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥癸拷锟教★拷一锟斤拷锟斤拷说锟斤拷LLM锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷直锟斤拷使锟斤拷PyTorch锟斤拷锟诫、使锟斤拷 VLLM / XInference / FastChat 锟饺匡拷埽锟揭诧拷锟斤拷锟绞癸拷锟� llama.cpp / chatglm.cpp / qwen.cpp 锟斤拷c++锟斤拷锟斤拷锟斤拷堋锟�

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷
- Greedy Search 贪锟斤拷锟斤拷锟斤拷锟斤拷式 锟斤拷锟斤拷锟斤拷前锟斤拷慕锟斤拷猓ｏ拷突岚达拷沾时锟斤拷叽锟斤拷锟斤拷筛锟斤拷省锟教帮拷锟斤拷锟绞斤拷岵伙拷锟窖★拷锟斤拷锟斤拷筛锟斤拷锟斤拷锟斤拷锟絫oken锟斤拷锟矫凤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷选锟斤拷锟斤拷锟斤拷锟斤拷锟绞ｏ拷锟斤拷锟侥ｏ拷突锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷馗锟斤拷锟斤拷锟斤拷郑锟揭伙拷锟绞碉拷锟接︼拷锟斤拷泻锟斤拷锟绞癸拷锟�
- Beam Search 锟斤拷贪锟斤拷锟斤拷式锟斤拷锟斤拷锟斤拷锟斤拷锟节ｏ拷beam search锟斤拷选锟斤拷锟斤拷锟斤拷锟斤拷锟絢锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷token时锟斤拷每锟斤拷前锟斤拷token锟斤拷锟斤拷锟斤拷锟斤拷k锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫撅拷锟斤拷k^2锟斤拷锟斤拷锟斤拷锟斤拷些锟斤拷锟斤拷锟斤拷选锟斤拷锟斤拷细锟斤拷锟斤拷锟斤拷锟絢锟斤拷锟斤拷锟斤拷锟捷癸拷锟街达拷锟斤拷锟饺ワ拷锟絢锟斤拷beam search锟姐法锟叫憋拷锟斤拷为beam_size
- Sample 锟斤拷锟斤拷锟斤拷锟斤拷锟绞斤拷锟斤拷锟斤拷沾时锟矫匡拷锟絫oken锟侥革拷锟绞诧拷锟斤拷一锟斤拷token锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟绞斤拷锟斤拷锟斤拷愿锟角匡拷锟斤拷锟侥壳帮拷锟斤拷锟斤拷锟斤拷锟斤拷煞锟绞斤拷锟�

1. 前锟斤拷

1.1 锟斤拷要锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

do_sample锟斤拷锟斤拷锟斤拷锟斤拷锟酵★拷锟角凤拷使锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟绞斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷为False锟斤拷锟斤拷使锟斤拷beam_search锟斤拷式
temperature锟斤拷锟斤拷锟节碉拷锟斤拷锟斤拷母锟斤拷锟斤拷锟斤拷锟斤拷锟绞轿拷锟�

$$
q_i=\frac{\exp(z_i/T)}{\sum_{j}\exp(z_j/T)}\
$$

锟接癸拷式锟斤拷锟皆匡拷锟斤拷锟斤拷锟斤拷锟絋取值为0锟斤拷锟斤拷效锟斤拷锟斤拷锟斤拷argmax锟斤拷锟斤拷时锟斤拷锟斤拷锟斤拷锟斤拷没锟斤拷锟斤拷锟斤拷裕锟饺≈滴拷锟斤拷锟斤拷锟绞憋拷咏锟斤拷锟饺∑斤拷锟斤拷锟揭伙拷锟絫emperature取值锟斤拷锟斤拷[0, 1]之锟戒。取值越锟斤拷锟斤拷锟叫э拷锟皆斤拷锟斤拷锟斤拷

锟斤拷锟斤拷锟斤拷蚀锟街伙拷锟斤拷锟饺凤拷锟斤拷源鸢福锟斤拷锟絋值锟斤拷锟斤拷为0锟斤拷锟斤拷之锟斤拷锟斤拷为锟斤拷锟斤拷0锟斤拷
top_k锟斤拷锟斤拷锟斤拷0锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷k锟斤拷锟斤拷锟斤拷锟斤拷锟侥斤拷锟斤拷薪锟斤拷胁锟斤拷锟斤拷锟絢越锟斤拷锟斤拷锟斤拷锟皆角匡拷锟皆叫∪凤拷锟斤拷锟皆角匡拷锟揭伙拷锟斤拷锟斤拷锟轿�20~100之锟戒。
- 实锟斤拷实锟斤拷锟叫匡拷锟斤拷锟饺达拷100锟斤拷始锟斤拷锟皆ｏ拷锟金步斤拷锟斤拷top_k直锟斤拷效锟斤拷锟斤到锟斤拷选锟�
top_p锟斤拷锟斤拷锟斤拷0锟侥革拷锟斤拷锟斤拷锟斤拷使锟斤拷锟叫憋拷锟斤拷锟角的斤拷锟斤拷母锟斤拷屎痛锟斤拷锟絧值锟斤拷p值越锟斤拷锟斤拷锟斤拷锟皆角匡拷锟皆叫∪凤拷锟斤拷锟皆角匡拷锟揭伙拷锟斤拷锟斤拷锟�0.7~0.95之锟戒。
- 实锟斤拷实锟斤拷锟叫匡拷锟斤拷锟饺达拷0.95锟斤拷始锟斤拷锟酵ｏ拷直锟斤拷效锟斤拷锟斤到锟斤拷选锟�
- top_p锟斤拷top_k锟斤拷锟斤拷效锟斤拷应锟斤拷锟饺碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟�
repetition_penalty锟斤拷锟斤拷锟节碉拷锟斤拷1.0锟侥革拷锟斤拷锟斤拷锟斤拷锟斤拷纬头锟斤拷馗锟絫oken锟斤拷默锟斤拷1.0锟斤拷锟斤拷没锟叫惩凤拷锟斤拷

1.2 KVCache

锟斤拷锟斤拷锟斤拷锟角斤拷锟斤拷锟斤拷锟皆回癸拷模锟酵碉拷锟斤拷锟斤拷锟角斤拷锟铰碉拷token锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷token锟侥癸拷锟教★拷锟斤拷么锟斤拷前锟斤拷token锟窖撅拷锟斤拷锟缴碉拷锟叫硷拷锟斤拷锟斤拷锟斤拷强锟斤拷锟街憋拷锟斤拷锟斤拷玫摹锟斤拷锟斤拷锟斤拷锟紸ttention锟结构锟斤拷说锟斤拷

$$
\text { Attention }(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V
$$

锟斤拷锟斤拷时锟斤拷Q锟角碉拷token tensor锟斤拷锟斤拷K锟斤拷V锟斤拷锟角帮拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷史token tensor锟侥筹拷锟斤拷锟叫ｏ拷锟斤拷锟終V锟角匡拷锟斤拷使锟斤拷前锟斤拷锟斤拷锟斤拷锟叫硷拷锟斤拷锟侥ｏ拷锟解部锟街的伙拷锟斤拷锟斤拷锟終VCache锟斤拷锟斤拷锟皆达拷占锟矫非筹拷锟睫达拷

2. VLLM锟斤拷锟�

锟斤拷址: https://github.com/vllm-project/vllm

vLLM锟斤拷一锟斤拷锟斤拷源锟侥达拷模锟斤拷锟斤拷锟斤拷锟斤拷锟劫匡拷埽锟酵拷锟絇agedAttention锟斤拷效锟截癸拷锟斤拷attention锟叫伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟绞碉拷锟斤拷吮锟紿uggingFace Transformers锟斤拷14-24锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

PagedAttention 锟斤拷 vLLM 锟侥猴拷锟侥硷拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟絃LM锟斤拷锟斤拷锟斤拷锟节达拷锟狡匡拷锟斤拷锟斤拷狻ｏ拷锟酵筹拷锟阶拷锟斤拷锟斤拷惴拷锟斤拷曰毓锟斤拷锟斤拷锟斤拷锟斤拷校锟斤拷锟揭拷锟斤拷锟斤拷锟斤拷锟斤拷锟絋oken锟斤拷注锟斤拷锟斤拷锟斤拷锟斤拷值锟斤拷锟斤拷锟芥储锟斤拷GPU锟节达拷锟叫ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷Token锟斤拷锟斤拷些锟斤拷锟斤拷募锟斤拷锟街碉拷锟斤拷锟酵拷锟斤拷锟斤拷锟轿狵V锟斤拷锟芥。

锟斤拷要锟斤拷锟斤拷
- 通锟斤拷PagedAttention锟斤拷 KV Cache 锟斤拷锟斤拷效锟斤拷锟斤拷
- 锟斤拷锟斤拷锟斤拷锟斤拷锟絚ontinus batching锟斤拷锟斤拷锟斤拷锟斤拷static batching
- 支锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷
- 支锟斤拷锟斤拷式锟斤拷锟�
- 锟斤拷锟斤拷 OpenAI 锟侥接口凤拷锟斤拷
- 锟斤拷 HuggingFace 模锟斤拷锟睫缝集锟斤拷

VLLM支锟街撅拷锟斤拷锟斤拷锟絃LM模锟酵碉拷锟斤拷锟斤拷锟斤拷锟劫★拷锟斤拷使锟斤拷锟斤拷锟铰的凤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷俣龋锟�

Continuous batching
- 锟斤拷实锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫ｏ拷一锟斤拷锟斤拷锟轿讹拷锟斤拷锟斤拷拥锟斤拷锟斤拷锟斤拷token锟斤拷锟饺匡拷锟斤拷锟斤拷锟杰达拷锟斤拷锟斤拷锟斤拷傻锟侥ｏ拷锟斤拷锟斤拷token锟斤拷锟斤拷锟斤拷锟揭诧拷艽锟斤拷锟絧ython锟斤拷锟斤拷锟斤拷锟斤拷锟叫ｏ拷锟斤拷痰锟斤拷锟斤拷谢锟饺达拷锟筋长锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷珊锟揭伙拷锟斤拷锟斤拷兀锟斤拷锟斤拷锟轿讹拷疟锟斤拷锟斤拷锟斤拷源锟斤拷锟斤拷锟斤拷锟絫oken锟斤拷GPU锟斤拷锟斤拷锟节讹拷锟斤拷锟斤拷锟斤拷胁锟斤拷锟斤拷锟斤拷朔选锟絚ontinous batching锟侥凤拷式锟斤拷锟斤拷锟斤拷每锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟接碉拷token锟斤拷锟斤拷锟斤拷锟斤拷效锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷
PagedAttention
- 锟斤拷锟斤拷时锟斤拷锟皆达拷占锟斤拷锟叫ｏ拷KVCache锟斤拷锟斤拷片锟斤拷锟斤拷锟截革拷锟斤拷录锟剿凤拷锟斤拷50%锟斤拷锟较碉拷锟皆存。VLLM锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷token锟斤拷锟斤拷锟斤拷锟斤拷锟街块，使每锟斤拷锟皆达拷锟节诧拷锟斤拷锟斤拷锟剿固讹拷锟斤拷锟饺碉拷tokens锟斤拷锟节斤拷锟斤拷Attention锟斤拷锟斤拷时锟斤拷VLLM锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟饺★拷锟終VCache锟斤拷锟斤拷锟姐。锟斤拷锟侥ｏ拷涂锟斤拷锟斤拷锟斤拷呒锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷模锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥碉拷址锟斤拷锟杰诧拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷诖锟斤拷思锟斤拷浅锟斤拷锟斤拷啤锟斤拷锟斤拷锟斤拷锟斤拷同一锟斤拷锟斤拷锟斤拷锟斤拷锟缴讹拷锟斤拷卮锟斤拷锟斤拷锟斤拷锟絍LLM锟结将锟斤拷同锟斤拷锟竭硷拷锟斤拷映锟斤拷为一锟斤拷锟斤拷锟斤拷锟介，锟金到斤拷省锟皆达拷锟斤拷锟斤拷锟斤拷碌锟斤拷锟斤拷谩锟�

LLM 大模型学习必知必会系列(十二)：VLLM性能飞跃部署实践：从推理加速到高效部署的全方位优化[更多内容：XInference/FastChat等框架]

值锟斤拷注锟斤拷锟斤拷牵锟絍LLM锟斤拷默锟较斤拷锟皆匡拷锟斤拷全锟斤拷锟皆达拷预锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷呋锟斤拷锟斤拷小锟斤拷锟斤拷锟斤拷锟劫度ｏ拷锟矫伙拷锟斤拷锟斤拷通锟斤拷锟斤拷锟斤拷 gpu_memory_utilization 锟斤拷锟狡伙拷锟斤拷锟叫★拷锟�

锟斤拷锟饺帮拷装VLLM锟斤拷

pip install vllm

import os
os.environ['VLLM_USE_MODELSCOPE'] = 'True'
from vllm import LLM, SamplingParams
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="qwen/Qwen-1_8B", trust_remote_code=True)
outputs = llm.generate(prompts, sampling_params)

#Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

注锟解，锟斤拷止锟斤拷锟斤拷锟侥碉拷锟斤拷写锟斤拷桑锟絍LLM锟斤拷Chat模锟酵碉拷锟斤拷锟斤拷支锟街ｏ拷模锟斤拷徒锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷猓拷锟绞碉拷式锟斤拷胁锟斤拷锟绞憋拷肟硷拷锟绞癸拷锟絊WIFT锟斤拷锟斤拷FastChat锟斤拷

LLM锟斤拷generate锟斤拷锟斤拷支锟斤拷直锟斤拷锟斤拷锟斤拷拼锟接好碉拷tokens(prompt_token_ids锟斤拷锟斤拷锟斤拷锟斤拷时锟斤拷要锟斤拷锟斤拷prompts锟斤拷锟斤拷)锟斤拷锟斤拷锟斤拷锟解部锟斤拷锟皆帮拷锟斤拷锟皆硷拷锟斤拷模锟斤拷锟斤拷锟狡达拷雍锟斤拷锟絍LLM锟斤拷SWIFT锟斤拷锟斤拷使锟斤拷锟斤拷锟斤拷锟街凤拷锟斤拷

锟斤拷锟斤拷锟斤拷锟铰斤拷锟斤拷锟斤拷锟角斤拷锟斤拷锟斤拷 AWQ锟斤拷锟斤拷锟斤拷VLLM直锟斤拷支锟街达拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷徒锟斤拷锟斤拷锟斤拷锟斤拷锟�

from vllm import LLM, SamplingParams
import os
import torch
os.environ['VLLM_USE_MODELSCOPE'] = 'True'

#Sample prompts.
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
#Create a sampling params object.
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

#Create an LLM.
llm = LLM(model="ticoAg/Qwen-1_8B-Chat-Int4-awq", quantization="AWQ", dtype=torch.float16, trust_remote_code=True)
#Generate texts from the prompts. The output is a list of RequestOutput objects
#that contain the prompt, generated text, and other information.
outputs = llm.generate(prompts, sampling_params)
#Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

VLLM锟劫凤拷锟侥碉拷锟斤拷锟皆查看锟斤拷锟斤拷锟斤拷

3.SWIFT

锟斤拷址锟斤拷 https://github.com/modelscope/swift/tree/main

SWIFT锟斤拷Scalable lightWeight Infrastructure for Fine-Tuning锟斤拷锟角伙拷锟斤拷PyTorch锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟戒即锟矫碉拷模锟斤拷微锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷堋锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷烁锟斤拷嗫磘uners锟斤拷锟斤拷LoRA锟斤拷QLoRA锟斤拷Adapter锟饺ｏ拷锟斤拷锟斤拷锟节猴拷锟斤拷ModelScope锟斤拷锟斤拷锟斤拷锟叫碉拷锟斤拷锟斤拷tuner ResTuning锟斤拷锟斤拷锟斤拷锟节此ｏ拷锟斤拷锟斤拷模态锟侥匡拷锟斤拷锟竭撅拷锟斤拷锟斤拷锟揭碉拷锟绞猴拷锟皆硷拷模锟酵的匡拷锟斤拷锟斤拷式锟斤拷

SWIFT锟斤拷tuners锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷Transformer锟结构锟斤拷模锟酵ｏ拷也锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟结构锟斤拷锟斤拷锟窖澳ｏ拷停锟斤拷锟斤拷锟揭伙拷写锟斤拷氪达拷锟斤拷锟轿拷锟侥ｏ拷停锟绞碉拷植锟斤拷锟斤拷锟叫э拷锟斤拷诖锟斤拷效锟斤拷时锟斤拷锟叫э拷锟窖碉拷锟斤拷锟斤拷獭锟�

SWIFT锟斤拷锟斤拷锟睫缝集锟缴碉拷ModelScope锟斤拷态系统锟叫ｏ拷锟斤拷通锟斤拷锟捷硷拷锟斤拷取锟斤拷模锟斤拷锟斤拷锟截★拷模锟斤拷训锟斤拷锟斤拷模锟斤拷锟斤拷锟斤拷锟斤拷模锟斤拷锟较达拷锟斤拷锟斤拷锟教★拷锟斤拷锟解，SWIFT锟斤拷PEFT锟斤拷全锟斤拷锟捷ｏ拷锟斤拷悉PEFT锟斤拷锟矫伙拷锟斤拷锟斤拷使锟斤拷SWIFT锟斤拷锟斤拷锟斤拷锟組odelScope锟斤拷模锟酵斤拷锟叫憋拷莸锟窖碉拷锟斤拷锟斤拷锟斤拷锟�

锟斤拷为ModelScope锟斤拷锟斤拷锟斤拷锟叫的匡拷源锟斤拷锟斤拷锟斤拷tuner ResTuning锟斤拷锟矫硷拷锟斤拷锟斤拷cv锟斤拷锟斤拷模态锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟较碉拷锟斤拷锟街わ拷锟斤拷锟窖碉拷锟叫э拷锟斤拷锟斤拷锟斤拷锟轿拷锟斤拷锟斤拷锟斤拷嗟憋拷锟斤拷锟斤拷锟铰ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟皆达拷锟绞�30%~60%锟斤拷为cv锟斤拷锟斤拷模态模锟酵碉拷训锟斤拷锟结供锟斤拷锟铰的凤拷式锟斤拷锟斤拷未锟斤拷锟斤拷应锟斤拷锟斤拷越锟斤拷越锟斤拷某锟斤拷锟斤拷稀锟�

SWIFT 锟斤拷锟斤拷锟揭拷锟斤拷锟斤拷锟斤拷裕锟�
- 锟竭憋拷SOTA锟斤拷锟皆碉拷Efficient Tuners锟斤拷锟斤拷锟节斤拷洗锟侥ｏ拷锟绞碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟揭碉拷锟斤拷钥锟斤拷希锟斤拷锟絉TX3080锟斤拷RTX3090锟斤拷RTX4090锟饺ｏ拷训锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷取锟矫较猴拷效锟斤拷
- 使锟斤拷ModelScope Hub锟斤拷Trainer锟斤拷锟斤拷锟斤拷transformers trainer锟结供锟斤拷支锟斤拷LLM模锟酵碉拷训锟斤拷锟斤拷锟斤拷支锟街斤拷训锟斤拷锟斤拷锟侥ｏ拷锟斤拷洗锟斤拷锟組odelScope Hub锟斤拷
- 锟斤拷锟斤拷锟叫碉拷模锟斤拷Examples锟斤拷锟斤拷锟斤拷锟斤拷糯锟侥ｏ拷锟斤拷峁╋拷锟窖碉拷锟斤拷疟锟斤拷锟斤拷锟斤拷锟斤拷疟锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟脚匡拷源锟斤拷锟捷硷拷锟结供锟斤拷预锟斤拷锟斤拷锟竭硷拷锟斤拷锟斤拷直锟斤拷锟斤拷锟斤拷使锟斤拷
- 支锟街斤拷锟芥化训锟斤拷锟斤拷锟斤拷锟斤拷

LLM 大模型学习必知必会系列(十二)：VLLM性能飞跃部署实践：从推理加速到高效部署的全方位优化[更多内容：XInference/FastChat等框架]

锟斤拷SWIFT锟叫ｏ拷锟斤拷锟斤拷支锟斤拷锟斤拷VLLM锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟街段★拷

pip install ms-swift[llm] openai

只锟斤拷要锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟酵匡拷锟斤拷使锟斤拷VLLM锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

swift infer --model_id_or_path qwen/Qwen-1_8B-Chat --max_new_tokens 128 --temperature 0.3 --top_p 0.7 --repetition_penalty 1.05 --do_sample true

也支锟斤拷锟节诧拷锟斤拷锟斤拷使锟斤拷VLLM锟斤拷

swift deploy --model_id_or_path qwen/Qwen-1_8B-Chat --max_new_tokens 128 --temperature 0.3 --top_p 0.7 --repetition_penalty 1.05 --do_sample true

锟斤拷锟矫ｏ拷

from openai import OpenAI
client = OpenAI(
    api_key='EMPTY',
    base_url='http://localhost:8000/v1',
)
model_type = client.models.list().data[0].id
print(f'model_type: {model_type}')

query = '锟姐江锟斤拷省锟斤拷锟斤拷锟斤拷锟斤拷?'
messages = [{
    'role': 'user',
    'content': query
}]
resp = client.chat.completions.create(
    model=model_type,
    messages=messages,
    seed=42)
response = resp.choices[0].message.content
print(f'query: {query}')
print(f'response: {response}')

#锟斤拷式
messages.append({'role': 'assistant', 'content': response})
query = '锟斤拷锟斤拷什么锟矫吃碉拷?'
messages.append({'role': 'user', 'content': query})
stream_resp = client.chat.completions.create(
    model=model_type,
    messages=messages,
    stream=True,
    seed=42)

print(f'query: {query}')
print('response: ', end='')
for chunk in stream_resp:
    print(chunk.choices[0].delta.content, end='', flush=True)
print()

"""Out[0]
model_type: qwen-7b-chat
query: 锟姐江锟斤拷省锟斤拷锟斤拷锟斤拷锟斤拷?
response: 锟姐江省锟斤拷省锟斤拷锟角猴拷锟斤拷锟叫★拷
query: 锟斤拷锟斤拷什么锟矫吃碉拷?
response: 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷食锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟姐、锟斤拷锟斤拷锟解、锟斤拷锟斤拷虾锟绞★拷锟叫伙拷童锟接硷拷锟饺★拷锟斤拷锟解，锟斤拷锟捷伙拷锟斤拷锟斤拷锟斤拷锟斤拷色小锟皆ｏ拷锟斤拷锟斤拷锟斤拷藕锟桔★拷锟斤拷锟斤拷小锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟饺★拷
"""

4.llama.cpp

llama.cpp锟斤拷使锟斤拷c++锟斤拷锟皆憋拷写锟侥讹拷llama系锟斤拷模锟酵斤拷锟叫革拷效锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥匡拷源锟解。锟矫匡拷使锟斤拷锟斤拷ggml锟阶诧拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟绞癸拷锟街帮拷锟揭拷锟斤拷饨玴ython锟斤拷weights转为ggml锟斤拷式锟斤拷gguf锟斤拷式锟斤拷锟斤拷使锟矫★拷锟斤拷llama.cpp锟斤拷锟狡ｏ拷锟斤拷锟叫硷拷锟斤拷ChatGLM模锟酵碉拷chatglm.cpp锟酵硷拷锟斤拷qwen模锟酵碉拷qwen.cpp锟斤拷mistral锟斤拷mistral.cpp锟斤拷

锟斤拷装锟斤拷锟斤拷锟斤拷

pip install modelscope

git clone --recursive https://github.com/QwenLM/qwen.cpp && cd qwen.cpp
cmake -B build
cmake --build build -j --config Release

锟斤拷锟斤拷模锟酵ｏ拷

from modelscope import snapshot_download
print(snapshot_download('qwen/Qwen-1_8B-Chat'))
#/mnt/workspace/.cache/modelscope/qwen/Qwen-1_8B-Chat

锟斤拷原始模锟斤拷转锟斤拷为ggml支锟街的革拷式锟斤拷

python3 qwen_cpp/convert.py -i /mnt/workspace/.cache/modelscope/qwen/Qwen-1_8B-Chat -t q4_0 -o qwen1_8b-ggml.bin
./build/bin/main -m qwen1_8b-ggml.bin --tiktoken /mnt/workspace/.cache/modelscope/qwen/Qwen-1_8B-Chat/qwen.tiktoken -p 锟斤拷锟�
#锟斤拷茫锟斤拷锟绞裁达拷铱锟斤拷园锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷锟斤拷锟铰斤拷锟斤拷锟斤拷锟角斤拷锟杰ｏ拷GGML锟斤拷锟绞猴拷锟斤拷CPU锟斤拷锟叫ｏ拷锟斤拷锟斤拷萍锟斤拷没锟斤拷锟紺PU锟斤拷锟斤拷锟叫伙拷锟皆碉拷锟斤拷锟斤拷锌锟斤拷锟絚pp锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟�

5.FastChat

FastChat Github锟斤拷址: https://github.com/lm-sys/FastChat
FastChat锟杰癸拷锟斤拷 https://github.com/lm-sys/FastChat/blob/main/docs/server_arch.md

FastChat锟斤拷一锟斤拷锟斤拷源锟斤拷锟斤拷锟解，锟斤拷锟斤拷锟斤拷模锟酵的分诧拷式锟斤拷锟斤拷实锟街ｏ拷锟斤拷锟结供锟斤拷OpenAI锟斤拷式锟斤拷RESTFul API锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷平台锟斤拷锟斤拷锟斤拷训锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷诖锟斤拷锟斤拷锟斤拷锟侥ｏ拷偷锟斤拷锟斤拷锟斤拷锟斤拷锟剿★拷

FastChat 锟侥猴拷锟侥癸拷锟杰帮拷锟斤拷锟斤拷
- 锟斤拷锟饺斤拷模锟酵碉拷训锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟诫（锟斤拷锟界，Vicuna锟斤拷MT-Bench锟斤拷锟斤拷
- 锟斤拷锟斤拷 Web UI 锟斤拷 OpenAI 锟斤拷锟斤拷 RESTful API 锟侥分诧拷式锟斤拷模锟酵凤拷锟斤拷系统

LLM 大模型学习必知必会系列(十二)：VLLM性能飞跃部署实践：从推理加速到高效部署的全方位优化[更多内容：XInference/FastChat等框架]

pip3 install "fschat[model_worker,webui]"
python3 -m fastchat.serve.controller

锟斤拷锟铰碉拷terminal锟斤拷锟斤拷锟斤拷锟斤拷

FASTCHAT_USE_MODELSCOPE=true python3 -m fastchat.serve.model_worker --model-path qwen/Qwen-1_8B-Chat --revision v1.0.0

之锟斤拷锟斤拷锟铰碉拷terminal锟叫匡拷锟斤拷锟斤拷锟叫斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟�:

python3 -m fastchat.serve.gradio_web_server

LLM 大模型学习必知必会系列(十二)：VLLM性能飞跃部署实践：从推理加速到高效部署的全方位优化[更多内容：XInference/FastChat等框架]

6.DeepSpeed

锟斤拷址锟斤拷 https://github.com/microsoft/DeepSpeed

锟斤拷址锟斤拷 https://www.deepspeed.ai/training/

Deepspeed锟斤拷锟叫匡拷芙锟斤拷锟� 锟斤拷 https://github.com/wzzzd/LLM_Learning_Note/blob/main/Parallel/deepspeed.md

Deepspeed锟斤拷微锟斤拷锟狡筹拷锟斤拷一锟斤拷锟斤拷源锟街诧拷式锟斤拷锟竭ｏ拷锟戒集锟斤拷锟剿分诧拷式训锟斤拷锟斤拷锟狡断★拷压锟斤拷锟饺革拷效模锟介。锟矫癸拷锟斤拷旨锟斤拷锟斤拷叽锟斤拷模模锟斤拷训锟斤拷锟斤拷效锟绞和匡拷锟斤拷展锟皆★拷锟斤拷通锟斤拷锟斤拷锟街硷拷锟斤拷锟街讹拷锟斤拷锟斤拷锟斤拷训锟斤拷锟斤拷锟斤拷锟斤拷模锟酵诧拷锟叫伙拷锟斤拷锟捷讹拷锟桔伙拷锟斤拷锟斤拷态锟斤拷锟斤拷锟斤拷锟脚★拷锟斤拷锟斤拷模式锟斤拷暇锟斤拷鹊取锟紻eepSpeed锟斤拷锟结供锟斤拷一些锟斤拷锟斤拷锟斤拷锟竭ｏ拷锟斤拷植锟绞窖碉拷锟斤拷锟斤拷锟斤拷锟斤拷诖锟斤拷呕锟斤拷锟侥ｏ拷锟窖癸拷锟斤拷龋锟斤拷园锟斤拷锟斤拷锟斤拷锟斤拷吒锟斤拷玫毓锟斤拷锟斤拷锟斤拷呕锟斤拷锟斤拷模锟斤拷锟窖把碉拷锟斤拷锟斤拷瘛４锟斤拷猓琩eepspeed锟斤拷锟斤拷pytorch锟斤拷锟斤拷锟斤拷只锟斤拷要锟斤拷锟睫改硷拷锟斤拷迁锟狡★拷 DeepSpeed锟窖撅拷锟斤拷锟斤拷锟斤拷锟斤拷模锟斤拷锟窖帮拷锟侥匡拷械玫锟斤拷锟接︼拷茫锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷汀锟酵硷拷锟斤拷锟洁、目锟斤拷锟斤拷取锟�

DeepSpeed锟斤拷锟斤拷Microsoft锟结供锟侥分诧拷式训锟斤拷锟斤拷锟竭ｏ拷旨锟斤拷支锟街革拷锟斤拷锟侥ｏ拷锟侥ｏ拷秃锟斤拷峁╋拷锟斤拷锟斤拷锟脚伙拷锟斤拷锟皆和癸拷锟竭★拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟饺ｏ拷DeepSpeed支锟街革拷锟斤拷锟侥ｏ拷锟侥ｏ拷秃锟斤拷峁╋拷锟斤拷锟斤拷锟脚伙拷锟斤拷锟皆和癸拷锟竭★拷锟斤拷锟叫ｏ拷锟斤拷要锟斤拷锟斤拷锟斤拷锟斤拷支锟街革拷锟斤拷锟侥ｏ拷锟侥ｏ拷汀锟斤拷峁╋拷烁锟斤拷锟斤拷锟脚伙拷锟斤拷锟皆和癸拷锟竭ｏ拷锟斤拷锟斤拷 ZeRO 锟斤拷 Offload 锟饺ｏ拷
- 锟斤拷 3D 锟斤拷锟叫伙拷实锟斤拷锟斤拷锟节诧拷锟斤拷模锟斤拷训锟斤拷锟斤拷 DeepSpeed 实锟斤拷锟斤拷锟斤拷锟街诧拷锟叫凤拷锟斤拷锟斤拷锟斤拷锟斤拷锟较ｏ拷ZeRO 支锟街碉拷锟斤拷锟捷诧拷锟叫ｏ拷锟斤拷水锟竭诧拷锟叫猴拷锟斤拷锟斤拷锟斤拷片模锟酵诧拷锟叫★拷3D 锟斤拷锟斤拷锟斤拷锟斤拷应锟剿诧拷同锟斤拷锟斤拷锟斤拷锟截碉拷锟斤拷锟斤拷锟斤拷支锟街撅拷锟斤拷锟斤拷锟节诧拷锟斤拷锟侥筹拷锟斤拷锟斤拷模锟酵ｏ拷同时实锟斤拷锟剿斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟皆达拷锟斤拷展锟皆猴拷锟斤拷锟斤拷锟斤拷锟斤拷展效锟绞★拷锟斤拷锟解，锟斤拷锟斤拷叩锟酵拷锟叫э拷锟绞癸拷没锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟睫的筹拷锟斤拷群锟斤拷锟斤拷锟斤拷 2-7 锟斤拷锟斤拷锟劫讹拷训锟斤拷锟斤拷锟斤拷十锟节诧拷锟斤拷锟斤拷模锟酵★拷
- ZeRO-Offload 使 GPU 锟斤拷锟斤拷锟杰癸拷训锟斤拷 10 锟斤拷锟斤拷锟侥ｏ拷停锟� 为锟斤拷同时锟斤拷锟斤拷 CPU 锟斤拷 GPU 锟节达拷锟斤拷训锟斤拷锟斤拷锟斤拷模锟酵ｏ拷锟斤拷锟斤拷锟斤拷展锟斤拷 ZeRO-2锟斤拷锟斤拷锟角碉拷锟矫伙拷锟斤拷使锟矫达拷锟叫碉拷锟斤拷英伟锟斤拷 V100 GPU 锟侥伙拷锟斤拷时锟斤拷锟斤拷锟斤拷锟节诧拷锟侥撅拷锟皆达拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫讹拷锟� 130 锟节革拷锟斤拷锟斤拷锟斤拷模锟酵ｏ拷模锟酵癸拷模锟斤拷展锟斤拷锟斤拷锟叫凤拷锟斤拷锟斤拷10锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫撅拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟剿癸拷锟斤拷使锟斤拷十锟节诧拷锟斤拷锟斤拷模锟斤拷训锟斤拷锟斤拷锟接达拷锟节伙拷锟斤拷锟斤拷锟斤拷为锟斤拷锟斤拷锟斤拷锟窖帮拷锟揭碉拷锟皆憋拷锟斤拷锟揭伙拷锟教斤拷锟斤拷锟斤拷锟斤拷锟矫碉拷模锟酵的达拷锟斤拷锟斤拷
- 通锟斤拷 DeepSpeed Sparse Attention 锟斤拷6锟斤拷锟劫讹拷执锟斤拷10锟斤拷锟斤拷锟斤拷锟斤拷锟叫ｏ拷 DeepSpeed锟结供锟斤拷稀锟斤拷 attention kernel 锟斤拷锟斤拷一锟街癸拷锟斤拷锟皆硷拷锟斤拷锟斤拷锟斤拷支锟街筹拷锟斤拷锟叫碉拷模锟斤拷锟斤拷锟诫，锟斤拷锟斤拷锟侥憋拷锟斤拷锟诫，图锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷搿ｏ拷刖拷锟侥筹拷锟斤拷 Transformer 锟斤拷龋锟斤拷锟街э拷值锟斤拷锟斤拷锟斤拷锟斤拷谐锟揭伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷诒锟斤拷锟斤拷嗟憋拷木锟斤拷锟斤拷禄锟斤拷锟斤拷锟� 6 锟斤拷锟斤拷执锟斤拷锟劫讹拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟铰碉拷稀锟斤拷实锟街匡拷 1.5锟紺3 锟斤拷锟斤拷锟斤拷锟解，锟斤拷锟角碉拷稀锟斤拷 kernel 锟斤拷锟街э拷锟较★拷锟斤拷式锟斤拷使锟矫伙拷锟杰癸拷通锟斤拷锟皆讹拷锟斤拷稀锟斤拷峁癸拷锟斤拷写锟斤拷隆锟�
- 1 锟斤拷锟斤拷 Adam 锟斤拷锟斤拷 5 锟斤拷通锟斤拷锟斤拷锟斤拷 Adam 锟斤拷一锟斤拷锟节达拷锟侥ｏ拷锟斤拷学习模锟斤拷训锟斤拷锟斤拷锟斤拷锟铰碉拷锟斤拷效锟侥ｏ拷也锟斤拷锟斤拷锟斤拷锟轿︼拷玫模锟斤拷呕锟斤拷锟斤拷锟饺伙拷锟斤拷锟斤拷锟斤拷锟酵拷锟叫э拷锟斤拷呕锟斤拷惴拷锟斤拷锟斤拷锟斤拷锟斤拷荨锟斤拷锟剿ｏ拷锟节匡拷锟借备锟斤拷锟叫分诧拷式锟斤拷展时锟斤拷通锟脚匡拷锟斤拷锟斤拷锟杰筹拷为瓶锟斤拷锟斤拷锟斤拷锟斤拷锟狡筹拷锟斤拷一锟斤拷 1 锟斤拷锟斤拷 Adam 锟斤拷锟姐法锟斤拷锟皆硷拷锟斤拷锟叫碉拷帧锟斤拷锟斤拷惴拷锟斤拷杉锟斤拷锟� 5 锟斤拷通锟斤拷锟斤拷锟斤拷同时实锟斤拷锟斤拷锟斤拷Adam锟斤拷锟狡碉拷锟斤拷锟斤拷锟绞★拷锟斤拷通锟斤拷锟斤拷锟睫的筹拷锟斤拷锟铰ｏ拷锟斤拷锟角观察到锟街诧拷式训锟斤拷锟劫讹拷锟斤拷锟斤拷锟斤拷 3.5 锟斤拷锟斤拷锟斤拷使锟矫革拷锟姐法锟斤拷锟斤拷锟斤拷展锟斤拷锟斤拷同锟斤拷锟酵碉拷 GPU 群锟斤拷锟斤拷锟斤拷锟界环锟斤拷锟斤拷
锟斤拷锟斤拷锟斤拷锟叫★拷锟�
- 锟斤拷锟紺PU锟斤拷锟斤拷锟斤拷llama.cpp 锟斤拷锟侥ｏ拷锟絠nt4锟斤拷锟斤拷锟斤拷锟斤拷训锟窖★拷锟�
- GPU锟斤拷锟斤拷锟斤拷微锟斤拷锟斤拷 DeepSpeed-FastGen 锟斤拷一锟斤拷锟矫碉拷选锟斤拷
- 锟街伙拷锟秸讹拷锟斤拷锟斤拷锟斤拷MLC LLM锟斤拷锟斤拷锟斤拷为锟斤拷选

LLM 大模型学习必知必会系列(十二)：VLLM性能飞跃部署实践：从推理加速到高效部署的全方位优化[更多内容：XInference/FastChat等框架]

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟阶拷锟斤拷牛锟酵∝硷拷斯锟斤拷锟斤拷埽锟斤拷锟斤拷峁┮恍╋拷锟截碉拷锟斤拷源锟斤拷锟斤拷锟斤拷锟斤拷锟铰ｏ拷锟斤拷鸦锟饺★拷亩锟斤拷锟�

小锟斤拷锟狡硷拷锟侥讹拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥斤拷为锟斤拷锟斤拷锟斤拷息锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷同锟斤拷锟睫观碉拷锟街な碉拷锟斤拷锟斤拷锟斤拷锟�

INF: 锟斤拷锟酵ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷营状态锟斤拷锟斤拷式锟斤拷营锟斤拷锟斤拷锟斤拷锟皆ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷; 锟斤拷戏锟斤拷锟斤拷

锟斤拷戏锟斤拷锟�

锟斤拷戏锟斤拷频

锟斤拷戏锟斤拷锟斤拷

锟斤拷戏锟筋动; 锟斤拷INF锟斤拷锟角讹拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷戏锟斤拷锟斤拷戏锟斤拷展锟斤拷锟剿黑白伙拷锟斤拷锟斤拷锟较凤拷锟今，革拷锟剿筹拷锟斤拷锟接撅拷锟较的硷拷锟斤拷锟斤拷锟介。锟斤拷