锟斤拷锟斤拷位锟矫ｏ拷锟斤拷页 > 锟斤拷锟斤拷锟教筹拷 > 锟教筹拷 > 通锟斤拷ORPO锟斤拷锟斤拷微锟斤拷 llama3锟斤拷模锟斤拷(Fine-tune Llama 3 with ORPO)

通锟斤拷ORPO锟斤拷锟斤拷微锟斤拷 llama3锟斤拷模锟斤拷(Fine-tune Llama 3 with ORPO)

锟斤拷源锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷|锟斤拷时锟戒：2024-04-23 10:50:52 |锟斤拷锟侥讹拷锟斤拷105锟斤拷|锟斤拷锟斤拷签锟斤拷 T A3 in 锟斤拷 |锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

1f45bd1e8577af66a05f5e3fadb0b29 通锟斤拷ORPO锟斤拷llama锟斤拷锟斤拷微锟斤拷前锟斤拷 ORPO锟斤拷一锟斤拷锟斤拷颖锟斤拷微锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷统锟侥监督微锟斤拷锟斤拷偏锟矫讹拷锟斤拷锥锟斤拷锟斤拷系锟揭伙拷锟斤拷锟斤拷锟斤拷小锟斤拷锟斤拷锟斤拷锟斤拷训锟斤拷锟斤拷锟斤拷募锟斤拷锟斤拷锟皆达拷锟绞憋拷洹ｏ拷锟斤拷猓拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟絆RPO锟节革拷锟斤拷模锟酵达拷小锟酵伙拷准锟斤拷锟斤拷锟叫讹拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟诫方锟斤拷锟斤拷锟节憋拷

通过ORPO技术微调 llama3大模型(Fine-tune Llama 3 with ORPO)

1f45bd1e8577af66a05f5e3fadb0b29

通锟斤拷ORPO锟斤拷llama锟斤拷锟斤拷微锟斤拷

前锟斤拷

ORPO锟斤拷一锟斤拷锟斤拷颖锟斤拷微锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷统锟侥监督微锟斤拷锟斤拷偏锟矫讹拷锟斤拷锥锟斤拷锟斤拷系锟揭伙拷锟斤拷锟斤拷锟斤拷小锟斤拷锟斤拷锟斤拷锟斤拷训锟斤拷锟斤拷锟斤拷募锟斤拷锟斤拷锟皆达拷锟绞憋拷洹ｏ拷锟斤拷猓拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟絆RPO锟节革拷锟斤拷模锟酵达拷小锟酵伙拷准锟斤拷锟斤拷锟叫讹拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟诫方锟斤拷锟斤拷锟节憋拷锟斤拷锟叫ｏ拷锟斤拷锟角斤拷使锟斤拷ORPO锟斤拷TRL锟斤拷锟斤拷微锟斤拷锟铰碉拷Llama 3 8B模锟酵★拷锟斤拷锟斤拷锟斤拷锟斤拷锟紾oogle Colab锟斤拷https://colab.research.google.com/drive/1eHNWg9gnaXErdAa8_mcvjMupbSS6rDvi?usp=sharing锟斤拷锟斤拷GitHub锟较碉拷LLM(https://github.com/mlabonne/llm-course)锟轿筹拷锟斤拷锟揭碉拷锟斤拷

?? ORPO

指锟斤拷锟斤拷锟斤拷锟狡拷枚锟斤拷锟斤拷墙锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷停锟絃LMs锟斤拷锟斤拷应锟截讹拷锟斤拷锟斤拷墓丶锟斤拷锟斤拷锟斤拷锟斤拷锟酵筹拷希锟斤拷锟斤拷婕帮拷锟揭伙拷锟斤拷锟阶段的癸拷锟教ｏ拷

锟斤拷指锟斤拷锟斤拷屑喽绞轿拷锟斤拷锟絊FT锟斤拷锟斤拷使模锟斤拷锟斤拷应目锟斤拷锟斤拷锟斤拷
锟斤拷锟斤拷锟洁反锟斤拷锟斤拷强锟斤拷学习锟斤拷RLHF锟斤拷锟斤拷直锟斤拷锟斤拷选锟脚伙拷锟斤拷DPO锟斤拷锟斤拷锟斤拷锟斤拷偏锟矫讹拷锟诫方锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷选锟斤拷应锟斤拷锟角憋拷锟杰撅拷锟斤拷应锟侥匡拷锟斤拷锟皆★拷

微锟斤拷图片_20240423001958

然锟斤拷锟斤拷锟叫撅拷锟斤拷员锟斤拷锟斤拷锟斤拷锟街凤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟皆★拷锟斤拷锟角监督微锟斤拷锟斤拷SFT锟斤拷锟斤拷锟斤拷锟斤拷效锟斤拷锟斤拷模锟斤拷锟斤拷应锟截讹拷锟斤拷锟斤拷锟斤拷也锟斤拷锟斤拷为什么锟斤拷要偏锟矫讹拷锟斤拷锥锟絉LHF锟斤拷锟斤拷锟斤拷锟杰伙拷迎锟斤拷锟斤拷筒锟斤拷芑锟接拷锟斤拷之锟斤拷锟斤拷实牟锟洁。

image

SFT锟斤拷锟斤拷锟叫ｏ拷锟斤拷锟杰伙拷迎锟斤拷锟斤拷锟斤拷锟斤拷实证 from ORPO锟斤拷锟斤拷

2024锟斤拷Hong锟斤拷Lee锟斤拷锟斤拷锟絆RPO通锟斤拷锟斤拷SFT锟斤拷RLHF统一为一锟斤拷锟斤拷锟斤拷训锟斤拷锟斤拷锟教ｏ拷为锟斤拷锟斤拷锟斤拷锟斤拷峁╋拷锟揭伙拷锟斤拷锟斤拷诺慕锟斤拷锟斤拷锟斤拷锟斤拷ORPO锟睫革拷锟剿憋拷准language model锟斤拷训锟斤拷目锟疥，锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷然锟斤拷失锟斤拷odds ratio(OR)锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷OR锟斤拷失锟皆诧拷锟杰伙拷迎锟斤拷锟斤拷锟绞╋拷锟斤拷锟斤拷锟轿拷某头锟斤拷锟酵憋拷哟锟斤拷锟斤拷芑锟接拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷模锟斤拷同时学习目锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷偏锟矫讹拷锟诫。

91e1091deacae95fb17f1b6995b94c2

ORPO锟窖撅拷锟斤拷锟斤拷要锟斤拷微锟斤拷锟斤拷锟叫得碉拷实锟街ｏ拷锟斤拷锟斤拷TRL锟斤拷Axolotl锟斤拷LLaMA-Factory锟斤拷锟斤拷锟斤拷一锟斤拷锟叫ｏ拷锟斤拷锟角斤拷锟斤拷锟斤拷锟斤拷锟绞癸拷锟絋RL锟斤拷锟叫诧拷锟斤拷锟斤拷

? 锟斤拷始通锟斤拷ORPO锟斤拷锟斤拷微锟斤拷

Llama3锟斤拷Meta锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷模锟酵ｏ拷LLM锟斤拷锟斤拷锟斤拷些模锟斤拷锟斤拷锟斤拷15锟斤拷锟斤拷token锟侥广泛锟斤拷锟捷硷拷锟斤拷训锟斤拷锟侥ｏ拷锟斤拷锟街拷拢锟絃lama2锟斤拷训锟斤拷锟斤拷锟捷硷拷为2锟斤拷锟斤拷token锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷模锟酵尺寸：一锟斤拷700锟节诧拷锟斤拷锟斤拷模锟酵猴拷一锟斤拷锟斤拷小锟斤拷80锟节诧拷锟斤拷锟斤拷模锟酵★拷700锟节诧拷锟斤拷锟斤拷模锟斤拷锟窖撅拷展示锟斤拷锟斤拷锟斤拷印锟斤拷锟斤拷痰锟斤拷锟斤拷埽锟斤拷锟組MLU锟斤拷准锟斤拷锟斤拷锟叫得凤拷为82锟斤拷锟斤拷HumanEval锟斤拷准锟斤拷锟斤拷锟叫得凤拷为81.7锟斤拷

Llama3模锟酵伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥筹拷锟饺ｏ拷锟斤拷锟缴达拷8192锟斤拷token锟斤拷Llama2为4096锟斤拷token锟斤拷锟斤拷锟斤拷锟揭匡拷锟斤拷通锟斤拷RoPE锟斤拷展锟斤拷32k锟斤拷锟斤拷锟解，锟斤拷些模锟斤拷使锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷128K-token锟绞伙拷锟斤拷锟斤拷路执锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷吮锟斤拷锟斤拷谋锟斤拷锟斤拷锟絫oken锟斤拷锟斤拷锟斤拷15%锟斤拷锟斤拷锟斤拷驶锟斤拷也锟斤拷锟斤拷锟剿达拷70锟节碉拷80锟节诧拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

image

ORPO锟斤拷要一锟斤拷偏锟斤拷锟斤拷锟捷硷拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷示锟斤拷一锟斤拷锟斤拷选锟斤拷拇鸢负锟揭伙拷锟斤拷锟斤拷芫锟斤拷拇鸢浮锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫ｏ拷锟斤拷锟角斤拷使锟斤拷mlabonne/orpo-dpo- mix-40k锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟铰革拷锟斤拷锟斤拷DPO锟斤拷锟捷硷拷锟斤拷隙锟斤拷傻锟斤拷锟斤拷菁锟斤拷锟�

argilla/distilabel-capybara-dpo-7k-binarized : 锟竭凤拷选锟斤拷拇锟� >=5锟斤拷2,882锟斤拷锟斤拷锟斤拷锟斤拷 https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized
argilla/distilabel-intel-orca-dpo-pairs : 锟竭凤拷选锟斤拷拇锟� >=9锟斤拷锟斤拷锟斤拷GSM8K锟叫ｏ拷2,299锟斤拷锟斤拷锟斤拷锟斤拷 https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs
argilla/ultrafeedback-binarized-preferences-cleaned : 锟竭凤拷选锟斤拷拇锟� >=5锟斤拷22,799锟斤拷锟斤拷锟斤拷锟斤拷 https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned
argilla/distilabel-math-preference-dpo : 锟竭凤拷选锟斤拷拇锟� >=9锟斤拷2,181锟斤拷锟斤拷锟斤拷锟斤拷 https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo
unalignment/toxic-dpo-v0.2 (541锟斤拷锟斤拷锟斤拷) https://huggingface.co/datasets/unalignment/toxic-dpo-v0.2
M4-ai/prm_dpo_pairs_cleaned (7,958锟斤拷锟斤拷锟斤拷) https://huggingface.co/datasets/M4-ai/prm_dpo_pairs_cleaned
jondurbin/truthy-dpo-v0.1 (1,016锟斤拷锟斤拷锟斤拷) https://huggingface.co/datasets/jondurbin/truthy-dpo-v0.1 锟斤拷谢argilla锟斤拷unalignment锟斤拷M4-ai锟斤拷jondurbin锟结供锟斤拷源锟斤拷锟捷硷拷锟斤拷

锟斤拷始锟斤拷装锟斤拷锟斤拷目猓�

??pip?install?-U?transformers?datasets?accelerate?peft?trl?bitsandbytes?wandb??

一锟斤拷锟斤拷装锟斤拷桑锟斤拷锟斤拷强锟斤拷缘锟斤拷锟斤拷要锟侥库，锟斤拷锟斤拷录锟斤拷W&B锟斤拷锟斤拷选锟斤拷锟斤拷

????import?gc??
????import?os??
??????
????import?torch??
????import?wandb??
????from?datasets?import?load_dataset??
????from?google.colab?import?userdata??
????from?peft?import?LoraConfig,?PeftModel,?prepare_model_for_kbit_training??
????from?transformers?import?(??
????????AutoModelForCausalLM,??
????????AutoTokenizer,??
????????BitsAndBytesConfig,??
????????TrainingArguments,??
????????pipeline,??
????)??
????from?trl?import?ORPOConfig,?ORPOTrainer,?setup_chat_format??
??????
????wb_token?=?userdata.get('wandb')??
????wandb.login(key=wb_token)??

锟斤拷锟斤拷锟斤拷锟揭伙拷锟斤拷锟铰碉拷GPU锟斤拷锟姐还应锟斤拷锟杰癸拷使锟斤拷Flash Attention锟斤拷锟斤拷锟芥换默锟较碉拷锟斤拷锟叫癸拷注实锟街ｏ拷锟斤拷一锟斤拷锟斤拷锟斤拷效锟侥凤拷式锟斤拷实锟街★拷

????if?torch.cuda.get_device_capability()[0]?>=?8:??
????????!pip?install?-qqq?flash-attn??
????????attn_implementation?=?"flash_attention_2"??
????????torch_dtype?=?torch.bfloat16??
????else:??
????????attn_implementation?=?"eager"??
????????torch_dtype?=?torch.float16??

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟角斤拷使锟斤拷bitsandbytes锟斤拷4位锟斤拷锟饺硷拷锟斤拷Llama 3 8B模锟酵★拷然锟斤拷锟斤拷锟斤拷使锟斤拷PEFT为QLoRA锟斤拷锟斤拷LoRA锟斤拷锟矫★拷锟揭伙拷使锟斤拷锟剿凤拷锟斤拷锟絪etup_chat_format()锟斤拷锟斤拷锟斤拷锟睫革拷模锟酵猴拷为ChatML支锟街的分达拷锟斤拷锟斤拷锟斤拷锟斤拷锟皆讹拷应锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷澹拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟狡ｏ拷锟斤拷锟斤拷锟斤拷模锟酵碉拷嵌锟斤拷锟侥达拷小锟斤拷匹锟斤拷锟铰的词伙拷锟斤拷锟叫★拷锟� 锟斤拷注锟解，锟斤拷锟斤拷要锟结交锟斤拷锟斤拷锟斤拷芊锟斤拷锟絤eta-llama/Meta-Llama-3-8B锟斤拷锟斤拷锟斤拷要锟斤拷录锟斤拷锟斤拷锟紿ugging Face锟剿伙拷锟斤拷锟斤拷锟竭ｏ拷锟斤拷锟斤拷约锟斤拷锟轿达拷锟秸碉拷模锟酵革拷锟斤拷锟斤拷锟斤拷NousResearch/Meta--Llama-3-8B锟斤拷

????#?Model??
????base_model?=?"meta-llama/Meta-Llama-3-8B"??
????new_model?=?"OrpoLlama-3-8B"??
??????
????#?QLoRA?config??
????bnb_config?=?BitsAndBytesConfig(??
????????load_in_4bit=True,??
????????bnb_4bit_quant_type="nf4",??
????????bnb_4bit_compute_dtype=torch_dtype,??
????????bnb_4bit_use_double_quant=True,??
????)??
??????
????#?LoRA?config??
????peft_config?=?LoraConfig(??
????????r=16,??
????????lora_alpha=32,??
????????lora_dropout=0.05,??
????????bias="none",??
????????task_type="CAUSAL_LM",??
????????target_modules=['up_proj',?'down_proj',?'gate_proj',?'k_proj',?'q_proj',?'v_proj',?'o_proj']??
????)??
??????
????#?Load?tokenizer??
????tokenizer?=?AutoTokenizer.from_pretrained(base_model)??
??????
????#?Load?model??
????model?=?AutoModelForCausalLM.from_pretrained(??
????????base_model,??
????????quantization_config=bnb_config,??
????????device_map="auto",??
????????attn_implementation=attn_implementation??
????)??
????model,?tokenizer?=?setup_chat_format(model,?tokenizer)??
????model?=?prepare_model_for_kbit_training(model)??

锟斤拷锟斤拷模锟斤拷锟窖撅拷准锟斤拷锟矫斤拷锟斤拷训锟斤拷锟斤拷锟斤拷锟角匡拷锟皆达拷锟斤拷锟斤拷锟捷硷拷锟斤拷锟斤拷锟角硷拷锟斤拷mlabonne/orpo-dpo-mix-40k锟斤拷锟斤拷使锟斤拷apply_chat_template()锟斤拷锟斤拷锟斤拷锟斤拷chosen锟斤拷锟酵★拷rejected锟斤拷锟斤拷转锟斤拷为ChatML锟斤拷式锟斤拷锟斤拷注锟解，锟斤拷只使锟斤拷锟斤拷1,000锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟捷硷拷锟斤拷锟斤拷为锟斤拷锟斤拷锟斤拷锟斤拷锟结花锟斤拷太锟斤拷时锟戒。

????dataset_name?=?"mlabonne/orpo-dpo-mix-40k"??
????dataset?=?load_dataset(dataset_name,?split="all")??
????dataset?=?dataset.shuffle(seed=42).select(range(10))??
??????
????def?format_chat_template(row):??
????????row["chosen"]?=?tokenizer.apply_chat_template(row["chosen"],?tokenize=False)??
????????row["rejected"]?=?tokenizer.apply_chat_template(row["rejected"],?tokenize=False)??
????????return?row??
??????
????dataset?=?dataset.map(??
????????format_chat_template,??
????????num_proc=?os.cpu_count(),??
????)??
????dataset?=?dataset.train_test_split(test_size=0.01)??

锟斤拷锟饺ｏ拷锟斤拷锟斤拷锟斤拷要锟斤拷锟斤拷一些锟斤拷锟斤拷锟斤拷锟斤拷学习锟绞ｏ拷锟诫传统锟斤拷SFT锟斤拷锟斤拷DPO锟斤拷龋锟絆RPO使锟矫碉拷学习锟绞非筹拷锟酵★拷锟斤拷锟街�8e-6锟斤拷锟斤拷原始锟斤拷锟侥ｏ拷锟斤拷锟铰讹拷应锟斤拷SFT锟斤拷学习锟斤拷1e-5锟斤拷DPO锟斤拷学习锟斤拷5e-6锟斤拷锟揭斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷微锟斤拷锟叫斤拷锟斤拷锟斤拷锟接碉拷锟斤拷约1e-6锟斤拷 beta锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫的诧拷锟斤拷锟斤拷锟斤拷默锟斤拷值为0.1锟斤拷锟斤拷锟斤拷原始锟斤拷锟侥碉拷一锟斤拷锟斤拷录锟斤拷示锟斤拷锟斤拷锟酵拷锟斤拷锟斤拷锟斤拷芯锟窖★拷锟斤拷锟斤拷锟� 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷蟪ざ群锟斤拷锟斤拷锟斤拷锟叫★拷锟斤拷锟斤拷锟斤拷锟轿拷锟斤拷锟斤拷芏锟斤拷使锟斤拷VRAM锟斤拷锟节达拷锟斤拷锟斤拷锟斤拷约为20 GB锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷拢锟斤拷锟斤拷墙锟斤拷锟侥ｏ拷徒锟斤拷锟�3-5锟斤拷锟斤拷锟节碉拷训锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟角斤拷锟斤拷锟�1锟斤拷锟斤拷锟节★拷锟斤拷锟斤拷锟斤拷强锟斤拷锟绞癸拷锟絆RPOTrainer锟斤拷训锟斤拷模锟酵ｏ拷锟斤拷锟戒当一锟斤拷锟斤拷装锟斤拷锟斤拷

????orpo_args?=?ORPOConfig(??
????????learning_rate=8e-6,??
????????beta=0.1,??
????????lr_scheduler_type="linear",??
????????max_length=1024,??
????????max_prompt_length=512,??
????????per_device_train_batch_size=2,??
????????per_device_eval_batch_size=2,??
????????gradient_accumulation_steps=4,??
????????optim="paged_adamw_8bit",??
????????num_train_epochs=1,??
????????evaluation_strategy="steps",??
????????eval_steps=0.2,??
????????logging_steps=1,??
????????warmup_steps=10,??
????????report_to="wandb",??
????????output_dir="./results/",??
????)??
??????
????trainer?=?ORPOTrainer(??
????????model=model,??
????????args=orpo_args,??
????????train_dataset=dataset["train"],??
????????eval_dataset=dataset["test"],??
????????peft_config=peft_config,??
????????tokenizer=tokenizer,??
????)??
????trainer.train()??
????trainer.save_model(new_model)??

锟斤拷L4 GPU锟较讹拷锟斤拷1000锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷模锟斤拷训锟斤拷锟斤拷约锟斤拷要2锟斤拷小时锟斤拷锟斤拷锟斤拷锟角查看W&B锟斤拷图锟斤拷

image

锟斤拷loss锟斤拷锟斤拷时锟斤拷锟杰伙拷迎锟斤拷锟斤拷筒锟斤拷芑锟接拷锟斤拷之锟斤拷牟锟斤拷觳拷锟斤拷锟斤拷裕锟狡斤拷锟斤拷呓锟斤拷准确锟饺分憋拷锟斤拷愿锟斤拷锟�0锟斤拷0.5锟斤拷

锟斤拷原始锟斤拷锟斤拷锟叫ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷 Anthropic/hh-rlhf 锟斤拷锟捷硷拷锟斤拷161k锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷训锟斤拷模锟酵斤拷锟斤拷锟斤拷10锟斤拷epochs锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷械锟绞憋拷锟揭拷锟斤拷枚唷ｏ拷锟斤拷腔锟斤拷锟絃lama3锟斤拷锟斤拷锟斤拷实锟介，锟斤拷锟斤拷锟窖好碉拷锟斤拷锟揭凤拷锟斤拷锟斤拷锟斤拷锟角碉拷锟斤拷志锟斤拷锟斤拷谢Jiwoo Hong锟斤拷锟斤拷

锟节憋拷锟教程的斤拷尾锟斤拷锟斤拷锟斤拷锟角斤拷QLoRA锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷秃喜锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷偷锟紿ugging Face Hub锟斤拷

????#?Flush?memory??
????del?trainer,?model??
????gc.collect()??
????torch.cuda.empty_cache()??
??????
????#?Reload?tokenizer?and?model??
????tokenizer?=?AutoTokenizer.from_pretrained(base_model)??
????model?=?AutoModelForCausalLM.from_pretrained(??
????????base_model,??
????????low_cpu_mem_usage=True,??
????????return_dict=True,??
????????torch_dtype=torch.float16,??
????????device_map="auto",??
????)??
????model,?tokenizer?=?setup_chat_format(model,?tokenizer)??
??????
????#?Merge?adapter?with?base?model??
????model?=?PeftModel.from_pretrained(model,?new_model)??
????model?=?model.merge_and_unload()??
??????
????model.push_to_hub(new_model,?use_temp_dir=False)??
????tokenizer.push_to_hub(new_model,?use_temp_dir=False)??

锟斤拷喜锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟絃lama3锟斤拷mlabonne/OrpoLlama-3-8B锟侥匡拷锟斤拷微锟斤拷锟斤拷锟斤拷锟斤拷锟绞癸拷锟斤拷锟斤拷Hugging Face Space锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷notebook锟斤拷锟斤拷锟斤拷锟皆硷拷锟斤拷实锟斤拷锟斤拷锟斤拷使锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷模锟斤拷训锟斤拷锟斤拷锟姐，锟斤拷锟斤拷W&B锟斤拷锟斤拷锟斤拷强锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟揭伙拷锟斤拷使锟斤拷LLM AutoEval锟斤拷Nous锟侥伙拷准锟斤拷锟斤拷锟阶硷拷锟较斤拷锟斤拷锟斤拷一些锟斤拷锟斤拷锟斤拷

image

锟斤拷锟角碉拷ORPO微锟斤拷实锟斤拷锟斤拷锟洁当锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷嘶锟斤拷锟侥ｏ拷锟斤拷锟矫匡拷锟斤拷锟阶硷拷锟斤拷锟斤拷系锟斤拷锟斤拷堋锟斤拷锟斤拷锟斤拷锟斤拷斯锟斤拷锟侥ｏ拷锟斤拷锟揭很匡拷锟斤拷锟斤拷味锟斤拷锟斤拷锟斤拷锟斤拷40k锟斤拷锟斤拷锟较斤拷锟斤拷微锟斤拷锟斤拷锟斤拷锟斤拷锟杰好的斤拷锟斤拷锟�

锟斤拷锟节匡拷源锟斤拷锟斤拷锟斤拷说锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟侥碉拷时锟教ｏ拷越锟斤拷越锟斤拷母锟斤拷锟斤拷锟斤拷锟斤拷锟饺拷锟侥ｏ拷捅锟斤拷锟斤拷锟斤拷锟斤拷锟皆达拷涂锟斤拷锟饺拷锟侥ｏ拷锟街拷锟侥诧拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫★拷锟斤拷锟轿拷锟斤拷腔锟饺★拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟杰碉拷锟斤拷要锟斤拷锟竭★拷

image

锟斤拷锟斤拷

锟斤拷锟斤拷篇锟教筹拷锟叫ｏ拷锟斤拷锟角斤拷锟斤拷锟斤拷ORPO锟姐法锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷谓锟絊FT锟斤拷锟洁督式微锟斤拷锟斤拷锟斤拷RLHF统一为锟斤拷一锟侥癸拷锟教★拷然锟斤拷锟斤拷锟斤拷使锟斤拷TRL锟斤拷Transformer Reinforcement Learning锟斤拷锟斤拷一锟斤拷锟斤拷锟狡碉拷偏锟斤拷锟斤拷锟捷硷拷锟较碉拷Llama3-8B锟斤拷锟斤拷微锟斤拷锟斤拷锟斤拷锟斤拷模锟斤拷展示锟斤拷锟斤拷锟剿癸拷锟斤拷慕锟斤拷锟斤拷锟斤拷突锟斤拷锟斤拷ORPO锟斤拷为锟铰碉拷微锟斤拷锟斤拷式锟斤拷潜锟斤拷锟斤拷

锟斤拷希锟斤拷锟斤拷锟斤拷邪锟斤拷锟斤拷锟斤拷锟斤拷萍锟斤拷锟斤拷锟斤拷锟紺olab锟绞记憋拷锟斤拷微锟斤拷锟斤拷锟皆硷拷锟斤拷Llama3模锟酵★拷锟节斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫ｏ拷锟斤拷锟角斤拷锟斤拷锟斤拷锟斤拷未锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷菁锟斤拷锟斤拷锟斤拷锟斤拷锟揭伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷拥牡恪�

锟斤拷锟斤拷锟斤拷虏慰锟斤拷裕锟�https://huggingface.co/blog/mlabonne/orpo-llama-3

小锟斤拷锟狡硷拷锟侥讹拷