锟斤拷站锟斤拷锟斤拷

锟斤拷锟斤拷位锟矫ｏ拷锟斤拷页 > 锟斤拷锟斤拷锟教筹拷 > 锟教筹拷 > 强锟斤拷学习锟绞硷拷之锟斤拷ACE:Off-PolicyActor-CriticwithCausality-AwareEntropyRegularization锟斤拷

强锟斤拷学习锟绞硷拷之锟斤拷ACE:Off-PolicyActor-CriticwithCausality-AwareEntropyRegularization锟斤拷

锟斤拷源锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷|锟斤拷时锟戒：2024-10-18 09:46:01 |锟斤拷锟侥讹拷锟斤拷137锟斤拷|锟斤拷 锟斤拷签锟斤拷 a T CTO AWA Ri rop Pyre S C ICY Causality AR 锟斤拷 |锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷锟斤拷锟斤拷戏

2024锟斤拷ICML锟斤拷锟铰ｏ拷ACE:Off-PolicyActor-CriticwithCausality-AwareEntropyRegularization锟斤拷锟斤拷

强锟斤拷学习锟绞硷拷之锟斤拷ACE:Off-PolicyActor-CriticwithCausality-AwareEntropyRegularization锟斤拷

前锟皆ｏ拷

锟斤拷锟斤拷锟斤拷锟斤拷锟藉华锟斤拷目锟斤拷锟斤拷锟节诧拷士师锟斤拷写锟斤拷锟斤拷锟铰ｏ拷锟斤拷目锟斤拷页为 ACE (ace-rl.github.io) 锟斤拷锟斤拷2024锟斤拷7锟铰凤拷锟斤拷锟斤拷ICML锟节匡拷

锟斤拷为锟斤拷锟斤拷锟斤拷冢锟斤拷锟绞抵伙拷锟斤拷遥锟斤拷锟揭拷锟斤拷憧硷拷锟揭伙拷锟斤拷锟斤拷锟斤拷目锟斤拷前锟斤拷募锟狡拷锟斤拷露锟斤拷锟斤拷痰锟�

锟斤拷锟斤拷锟斤拷为强锟斤拷学习锟绞记碉拷5篇

锟斤拷锟侥筹拷锟洁辑锟斤拷2024.10.5锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟绞憋拷洌拷锟斤拷锟斤拷耍锟角帮拷锟叫达拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟�

CSDN锟斤拷页锟斤拷 https://blog.csdn.net/rvdgdsva

锟斤拷锟斤拷园锟斤拷页锟斤拷 https://www.cnblogs.com/hassle

锟斤拷锟斤拷园锟斤拷锟斤拷锟斤拷锟接ｏ拷

锟斤拷锟斤拷一锟斤拷

锟斤拷篇强锟斤拷学习锟斤拷锟斤拷锟斤拷要锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷为 ACE 锟斤拷锟姐法锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷为 Off-Policy Actor-Critic with Causality-Aware Entropy Regularization 锟斤拷锟斤拷通锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟较碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷锟斤拷诓锟酵拷锟斤拷锟轿拷锟斤拷系牟锟狡斤拷锟教斤拷锟斤拷锟斤拷猓硷拷诟慕锟角匡拷锟窖帮拷锟阶拷锟�1锟斤拷锟斤拷探锟斤拷效锟绞猴拷锟斤拷锟斤拷效锟绞碉拷锟斤拷锟解，锟截憋拷锟斤拷锟节革拷维锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫的憋拷锟街★拷

锟斤拷注锟斤拷1锟斤拷: 强锟斤拷学习锟斤拷锟斤拷锟斤拷一篇锟酵癸拷锟斤拷

锟斤拷锟斤拷摘要

锟斤拷policy锟斤拷注锟斤拷2锟斤拷学习锟斤拷锟斤拷锟叫ｏ拷锟斤拷同原始锟斤拷为锟侥诧拷同锟斤拷锟藉被锟斤拷前锟斤拷model-free RL 锟姐法锟斤拷锟斤拷锟接★拷锟斤拷锟斤拷锟斤拷一锟斤拷锟解，锟斤拷锟斤拷探锟斤拷锟剿诧拷同锟叫讹拷维锟饺和斤拷锟斤拷之锟斤拷锟斤拷锟斤拷锟斤拷系锟斤拷锟斤拷锟斤拷锟斤拷训锟斤拷锟斤拷锟斤拷锟叫革拷锟斤拷原始锟斤拷为锟斤拷锟斤拷要锟皆★拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟较碉拷锟街拷亍锟阶拷锟�3锟斤拷锟筋（causality-aware entropy term锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷效锟斤拷识锟斤拷锟斤拷锟饺匡拷锟角撅拷锟叫革拷潜锟斤拷影锟斤拷锟斤拷锟轿拷锟斤拷锟绞碉拷指锟叫э拷锟教斤拷锟斤拷锟斤拷锟斤拷猓拷朔锟街癸拷锟斤拷裙锟阶拷囟锟斤拷锟皆硷拷锟轿拷锟斤拷锟斤拷欠锟斤拷锟斤拷锟斤拷荻锟斤拷锟斤拷锟斤拷锟斤拷锟絞radientdormancyphenomenon锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟矫伙拷锟狡ｏ拷锟皆斤拷一锟斤拷锟斤拷强锟斤拷锟角凤拷锟斤拷锟斤拷锟斤拷效锟皆★拷锟斤拷锟斤拷模锟斤拷RL锟斤拷锟斤拷锟斤拷龋锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟姐法 ACE :Off-policy A ctor-criticwith C ausality-aware E ntropyregularization锟斤拷锟节匡拷越7锟斤拷锟斤拷锟�29锟街诧拷同锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷示锟斤拷实锟斤拷锟皆碉拷锟斤拷锟斤拷锟斤拷锟狡ｏ拷锟斤拷强锟斤拷锟斤拷锟斤拷锟角凤拷锟斤拷锟斤拷锟斤拷效锟皆★拷锟洁功锟斤拷锟皆和革拷效锟斤拷锟斤拷锟斤拷效锟绞★拷 锟斤拷准锟斤拷锟皆斤拷锟斤拷锟斤拷锟狡碉拷锟斤拷锟�https://ace-rl.github.io/锟较伙拷谩锟�

锟斤拷注锟斤拷2锟斤拷: 强锟斤拷学习锟姐法锟斤拷on-policy锟斤拷off-policy

锟斤拷注锟斤拷3锟斤拷: 锟斤拷锟斤拷锟� RL锟斤拷锟斤拷Soft Q-Learning锟斤拷SAC - 知锟斤拷

锟斤拷锟斤拷锟斤拷要锟斤拷锟阶ｏ拷

锟斤拷1锟斤拷 锟斤拷锟斤拷锟较碉拷锟斤拷锟� 锟斤拷通锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟�-锟斤拷锟斤拷锟结构模锟酵ｏ拷锟斤拷锟斤拷锟斤拷同锟斤拷锟斤拷维锟饺ｏ拷锟斤拷原始锟斤拷为锟斤拷锟皆斤拷锟斤拷锟斤拷影锟斤拷锟叫★拷锟斤拷锟轿拷锟斤拷锟斤拷权锟截★拷锟斤拷锟斤拷锟斤拷些权锟截凤拷映锟斤拷每锟斤拷锟斤拷锟斤拷维锟斤拷锟节诧拷同学习锟阶段碉拷锟斤拷锟斤拷锟揭拷浴锟�

锟斤拷锟斤拷锟斤拷锟斤拷锟侥斤拷锟斤拷原锟斤拷锟角ｏ拷锟斤拷锟斤拷一锟斤拷锟津单碉拷锟斤拷锟接ｏ拷一锟斤拷锟斤拷械锟斤拷锟斤拷锟接︼拷锟窖帮拷锟斤拷锟斤拷直鄄锟阶プ★拷锟斤拷澹伙拷锟阶拷锟斤拷锟阶拷频锟窖帮拷直鄢锟斤拷锟斤拷锟斤拷锟侥匡拷锟斤拷锟剿讹拷锟斤拷锟斤拷锟较★拷锟斤拷耍锟斤拷诓锟斤拷锟窖帮拷牟锟酵拷锥锟角匡拷锟斤拷锟斤拷锟斤拷锟揭拷锟皆硷拷锟轿拷锟教斤拷锟斤拷锟� 锟斤拷锟斤拷锟斤拷要锟侥★拷锟斤拷探锟斤拷锟斤拷锟斤拷锟叫匡拷锟斤拷锟阶拷锟斤拷锟皆硷拷锟轿拷锟斤拷锟斤拷约锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟矫匡拷锟斤拷锥味曰锟斤拷锟皆硷拷锟轿拷锟窖帮拷锟斤拷佣锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟剿讹拷锟斤拷锟斤拷锟叫э拷省锟�

锟剿达拷锟缴癸拷学习锟斤拷锟斤拷锟较ｏ拷

锟斤拷2锟斤拷 锟斤拷锟斤拷锟斤拷锟斤拷锟� 锟斤拷锟斤拷锟斤拷锟斤拷锟角匡拷锟窖帮拷锟杰的伙拷锟斤拷锟较ｏ拷锟斤拷SAC锟姐法锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 锟斤拷锟斤拷锟饺拷锟斤拷锟斤拷锟斤拷锟斤拷锟� 锟斤拷锟诫传统锟斤拷锟斤拷锟津化诧拷同锟斤拷锟斤拷一锟斤拷锟斤拷莞锟斤拷锟皆硷拷锟轿拷锟斤拷锟斤拷权锟截讹拷态锟斤拷锟斤拷锟斤拷强锟斤拷锟斤拷锟斤拷要锟斤拷为锟斤拷探锟斤拷锟斤拷锟斤拷锟劫对诧拷锟斤拷要锟斤拷为锟斤拷探锟斤拷锟斤拷

锟斤拷锟斤拷锟斤拷锟斤拷锟侥斤拷锟斤拷原锟斤拷锟角ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟�-锟斤拷锟斤拷锟结构模锟斤拷锟斤拷锟斤拷锟斤拷锟叫讹拷锟秸硷拷锟较碉拷锟斤拷锟饺拷锟�(causal weights)锟斤拷锟斤拷锟饺拷鼗锟斤拷锟斤拷锟絘gent锟斤拷锟叫革拷锟斤拷效锟斤拷探锟斤拷锟斤拷 锟斤拷锟斤拷锟斤拷锟斤拷锟饺拷亟洗锟侥讹拷锟斤拷维锟饺斤拷锟斤拷探锟斤拷锟斤拷锟斤拷锟斤拷锟皆斤拷锟斤拷锟斤拷锟斤拷要锟皆革拷锟襟，诧拷锟斤拷锟劫讹拷锟斤拷锟饺拷亟锟叫★拷锟斤拷锟轿拷鹊锟教� 锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷目锟斤拷缺锟斤拷锟皆诧拷同学习锟阶讹拷原始锟斤拷为之锟斤拷锟斤拷锟斤拷锟斤拷锟揭拷缘锟斤拷锟绞讹拷锟斤拷锟斤拷艿锟斤拷碌锟叫э拷锟教斤拷锟斤拷锟轿拷私锟斤拷锟斤拷一锟斤拷锟狡ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟饺拷丶锟饺拷牟锟斤拷锟斤拷锟斤拷锟轿拷锟斤拷锟斤拷系锟斤拷知锟斤拷锟斤拷锟斤拷锟侥匡拷辏拷锟叫э拷丶锟角匡拷硕锟斤拷锟揭硷拷锟轿拷锟教斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷烁锟斤拷锟叫э拷锟教斤拷锟斤拷锟�

锟剿达拷锟缴癸拷学习锟斤拷锟斤拷锟较ｏ拷

锟斤拷3锟斤拷 锟捷度★拷锟斤拷锟竭★拷锟斤拷锟斤拷Gradient Dormancy锟斤拷 锟斤拷锟斤拷锟侥观察到锟斤拷模锟斤拷训锟斤拷时锟斤拷些锟捷度伙拷锟斤拷某些锟阶段诧拷锟斤拷跃锟斤拷锟斤拷锟斤拷锟斤拷锟竭★拷锟斤拷锟斤拷为锟剿凤拷止模锟酵癸拷锟饺癸拷注某些原始锟斤拷为锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 锟捷讹拷锟斤拷锟竭碉拷锟斤拷锟斤拷锟斤拷没锟斤拷锟� 锟斤拷锟矫伙拷锟斤拷通锟斤拷锟斤拷锟斤拷锟皆地讹拷模锟酵斤拷锟斤拷锟脚讹拷锟斤拷reset锟斤拷锟斤拷锟斤拷锟斤拷模锟斤拷锟斤拷锟斤拷植锟斤拷锟斤拷牛锟斤拷俳锟斤拷锟斤拷惴猴拷锟教斤拷锟斤拷锟�

锟斤拷锟斤拷锟斤拷锟斤拷锟侥斤拷锟斤拷原锟斤拷锟角ｏ拷锟矫伙拷锟斤拷通锟斤拷一锟斤拷锟斤拷锟捷讹拷锟斤拷锟竭程度撅拷锟斤拷锟斤拷锟斤拷锟截硷拷歇锟皆地革拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷纭ｏ拷锟斤拷锟斤拷锟斤拷系锟斤拷知探锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷颖锟斤拷锟斤拷锟矫伙拷锟斤拷锟斤拷锟较ｏ拷旨锟节促斤拷锟斤拷锟斤拷效锟斤拷锟斤拷锟斤拷效锟斤拷探锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟杰★拷

通锟斤拷锟节讹拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷械锟绞碉拷椋珹CE 展示锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷强锟斤拷学习锟姐法锟斤拷锟斤拷SAC锟斤拷TD3锟斤拷锟侥憋拷锟街ｏ拷

  • 29锟斤拷锟斤拷同锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 锟斤拷锟斤拷锟斤拷 Meta-World锟斤拷12锟斤拷锟斤拷锟今）★拷DMControl锟斤拷5锟斤拷锟斤拷锟今）★拷Dexterous Hand锟斤拷3锟斤拷锟斤拷锟今）猴拷锟斤拷锟斤拷稀锟借奖锟斤拷锟斤拷锟斤拷6锟斤拷锟斤拷锟今）★拷
  • 实锟斤拷锟斤拷 锟斤拷锟斤拷锟斤拷ACE 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫讹拷锟斤到锟剿革拷锟矫碉拷锟斤拷锟斤拷效锟绞和革拷锟竭碉拷锟斤拷锟斤拷锟斤拷锟杰★拷锟斤拷锟界，锟节革拷锟接碉拷稀锟借奖锟斤拷锟斤拷锟斤拷锟叫ｏ拷ACE 凭锟斤拷锟斤拷锟斤拷锟饺拷锟斤拷锟斤拷锟斤拷锟教斤拷锟斤拷锟斤拷裕锟斤拷锟斤拷锟斤拷锟皆斤拷锟� SAC 锟斤拷 TD3 锟斤拷锟斤拷锟斤拷锟姐法锟斤拷

锟斤拷锟斤拷锟叫的对憋拷实锟斤拷图锟斤拷锟斤拷示锟斤拷 ACE 锟节讹拷锟斤拷锟斤拷锟斤拷锟铰碉拷锟斤拷锟斤拷锟斤拷锟狡ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷 稀锟借奖锟斤拷锟酵革拷维锟斤拷锟斤拷锟斤拷 锟叫ｏ拷ACE 凭锟斤拷锟斤拷探锟斤拷效锟绞碉拷锟斤拷锟斤拷锟斤拷锟杰革拷锟斤拷锏斤拷锟斤拷挪锟斤拷浴锟�

锟斤拷锟侥达拷锟斤拷锟斤拷

锟斤拷ACE原锟斤拷锟侥的碉拷21页锟斤拷锟斤拷锟斤拷锟斤拷锟接︼拷锟叫达拷锟斤拷锟狡拷模锟斤拷锟斤拷锟斤拷铱锟斤拷撕镁玫拇锟斤拷锟饺ワ拷锟斤拷锟斤拷锟�

锟斤拷锟斤拷说实锟斤拷锟斤拷伪锟斤拷锟斤拷锟叫癸拷锟斤拷锟侥ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷械锟斤拷锟斤拷一锟斤拷锟斤拷

锟斤拷锟斤拷一锟斤拷强锟斤拷学习锟斤拷RL锟斤拷锟姐法锟侥匡拷埽锟斤拷锟斤拷锟斤拷锟揭伙拷锟斤拷锟斤拷锟斤拷锟斤拷贫希锟紺ausal Discovery锟斤拷锟斤拷锟斤拷锟斤拷裕锟絆ff-policy锟斤拷Actor-Critic锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟角讹拷每锟斤拷模锟介及锟斤拷锟斤拷锟斤拷锟剿碉拷锟斤拷锟�

1. 锟斤拷始锟斤拷模锟斤拷

  • Q锟斤拷锟斤拷 ( \(Q_\phi\) ) 锟斤拷锟斤拷锟节癸拷锟狡讹拷锟斤拷锟斤拷值锟斤拷(\phi) 锟斤拷权锟截诧拷锟斤拷锟斤拷
  • 锟斤拷锟斤拷锟斤拷锟斤拷 ( $\pi_\theta $) 锟斤拷锟斤拷锟斤拷锟斤拷锟缴讹拷锟斤拷锟斤拷锟皆ｏ拷(\theta) 锟斤拷锟斤拷权锟截★拷
  • 锟截放伙拷锟斤拷锟斤拷 ($ D$ ) 锟斤拷锟芥储锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟捷ｏ拷锟皆憋拷锟斤拷胁锟斤拷锟斤拷锟�
  • 锟街诧拷锟斤拷锟斤拷锟斤拷 ( $D_c $) 锟斤拷锟芥储锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥局诧拷锟斤拷锟捷★拷
  • 锟斤拷锟饺拷鼐锟斤拷锟� ($ B_{a \rightarrow r|s} $) 锟斤拷锟斤拷锟节诧拷捉锟斤拷锟斤拷锟诫奖锟斤拷之锟斤拷锟斤拷锟斤拷锟斤拷系锟斤拷
  • 锟脚讹拷锟斤拷锟斤拷 ( \(f\) ) 锟斤拷锟斤拷锟节对诧拷锟皆斤拷锟斤拷微小锟脚讹拷锟斤拷锟斤拷锟斤拷探锟斤拷锟斤拷

2. 锟斤拷锟斤拷锟斤拷锟侥ｏ拷锟�

  • 每 ( $$I$$ ) 锟斤拷锟斤拷锟斤拷 锟斤拷
    • 锟斤拷锟斤拷锟斤拷锟斤拷 锟斤拷锟接局诧拷锟斤拷锟斤拷锟斤拷 ( \(D_c\) ) 锟叫筹拷锟斤拷 ( \(N_c\) ) 锟斤拷转锟狡★拷
    • 锟斤拷锟斤拷锟斤拷锟饺拷鼐锟斤拷锟� 锟斤拷锟斤拷锟斤拷 ($ B_{a \rightarrow r|s}$ )锟斤拷锟斤拷锟节凤拷映锟斤拷前锟斤拷锟皆和斤拷锟斤拷之锟斤拷锟斤拷锟斤拷锟斤拷系锟斤拷

3. 锟斤拷锟斤拷锟脚伙拷模锟斤拷

  • 每锟斤拷锟捷度诧拷锟斤拷 锟斤拷
    • 锟斤拷锟斤拷锟斤拷锟斤拷 锟斤拷锟斤拷锟截放伙拷锟斤拷锟斤拷 ( \(D\) ) 锟叫筹拷锟斤拷 ($ N$ ) 锟斤拷转锟狡★拷
    • 锟斤拷锟斤拷锟斤拷锟斤拷锟绞讹拷锟� ( \(H_c(\pi(\cdot|s))\) ) 锟斤拷锟斤拷锟斤拷锟节革拷锟斤拷状态锟铰诧拷锟皆碉拷锟斤拷锟斤拷院锟饺凤拷锟斤拷裕锟斤拷锟斤拷锟斤拷薷牟锟斤拷浴锟�
    • 目锟斤拷 Q 值锟斤拷锟斤拷 锟斤拷锟斤拷锟斤拷目锟斤拷 Q 值锟斤拷锟斤拷锟斤拷训锟斤拷 Q 锟斤拷锟界。
    • 锟斤拷锟斤拷 Q 锟斤拷锟斤拷 锟斤拷锟斤拷锟斤拷预锟斤拷锟� Q 值锟斤拷目锟斤拷 Q 值之锟斤拷锟斤拷锟筋。
    • 锟斤拷锟铰诧拷锟斤拷锟斤拷锟斤拷 锟斤拷锟斤拷蠡锟角白刺拷碌锟� Q 值锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷妗�

4. 锟斤拷锟矫伙拷锟斤拷模锟斤拷

  • 每锟斤拷锟斤拷锟矫硷拷锟� 锟斤拷
    • 锟斤拷锟斤拷锟捷讹拷锟斤拷锟斤拷锟斤拷 ( $\beta_\gamma $) 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟皆革拷锟铰碉拷影锟斤拷潭取锟�
    • 锟斤拷始锟斤拷锟斤拷锟斤拷锟斤拷锟� 锟斤拷为锟铰的诧拷锟皆革拷锟斤拷准锟斤拷锟斤拷始权锟斤拷 ( $\phi_i $)锟斤拷
    • 锟斤拷锟斤拷锟矫诧拷锟皆猴拷 Q 锟斤拷锟斤拷 锟斤拷锟斤拷锟斤拷锟斤拷锟饺拷亟锟斤拷锟狡斤拷锟斤拷锟斤拷拢锟斤拷锟斤拷锟绞碉拷指锟斤拷榷锟斤拷锟斤拷呕锟斤拷锟�
    • 锟斤拷锟矫诧拷锟皆猴拷 Q 锟脚伙拷锟斤拷 锟斤拷锟斤拷锟斤拷锟斤拷时锟斤拷锟阶刺拷锟斤拷员锟斤拷锟斤拷锟铰碉拷学习锟斤拷锟教★拷

锟斤拷锟斤拷源锟斤拷锟斤拷锟斤拷锟斤拷

源锟斤拷锟斤拷锟斤拷千锟斤拷锟截ｏ拷锟斤拷锟斤拷只锟斤拷锟斤拷锟斤拷main_casual锟斤拷锟斤拷牟锟斤拷执锟斤拷耄拷锟斤拷锟缴撅拷锟斤拷撕艽锟揭伙拷锟斤拷执锟斤拷锟斤拷员锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 

def train_loop(config, msg = "default"):
    # Agent
    agent = ACE_agent(env.observation_space.shape[0], env.action_space, config)

    memory = ReplayMemory(config.replay_size, config.seed)
    local_buffer = ReplayMemory(config.causal_sample_size, config.seed)

    for i_episode in itertools.count(1):
        done = False

        state = env.reset()
        while not done:
            if config.start_steps > total_numsteps:
                action = env.action_space.sample()  # Sample random action
            else:
                action = agent.select_action(state)  # Sample action from policy

            if len(memory) > config.batch_size:
                for i in range(config.updates_per_step):
                    #* Update parameters of causal weight
                    if (total_numsteps % config.causal_sample_interval == 0) and (len(local_buffer)>=config.causal_sample_size):
                        causal_weight, causal_computing_time = get_sa2r_weight(env, local_buffer, agent, sample_size=config.causal_sample_size, causal_method='DirectLiNGAM')
                        print("Current Causal Weight is: ",causal_weight)
                        
                    dormant_metrics = {}
                    # Update parameters of all the networks
                    critic_1_loss, critic_2_loss, policy_loss, ent_loss, alpha, q_sac, dormant_metrics = agent.update_parameters(memory, causal_weight,config.batch_size, updates)

                    updates += 1
            next_state, reward, done, info = env.step(action) # Step
            total_numsteps += 1
            episode_steps += 1
            episode_reward += reward

            #* Ignore the "done" signal if it comes from hitting the time horizon.
            if '_max_episode_steps' in dir(env):  
                mask = 1 if episode_steps == env._max_episode_steps else float(not done)
            elif 'max_path_length' in dir(env):
                mask = 1 if episode_steps == env.max_path_length else float(not done)
            else: 
                mask = 1 if episode_steps == 1000 else float(not done)

            memory.push(state, action, reward, next_state, mask) # Append transition to memory
            local_buffer.push(state, action, reward, next_state, mask) # Append transition to local_buffer
            state = next_state

        if total_numsteps > config.num_steps:
            break

        # test agent
        if i_episode % config.eval_interval == 0 and config.eval is True:
            eval_reward_list = []
            for _  in range(config.eval_episodes):
                state = env.reset()
                episode_reward = []
                done = False
                while not done:
                    action = agent.select_action(state, evaluate=True)
                    next_state, reward, done, info = env.step(action)
                    state = next_state
                    episode_reward.append(reward)
                eval_reward_list.append(sum(episode_reward))

            avg_reward = np.average(eval_reward_list)
          
    env.close()

锟斤拷锟斤拷锟斤拷锟教斤拷锟斤拷

  1. 锟斤拷始锟斤拷 :

    • 通锟斤拷锟斤拷锟斤拷锟侥硷拷 config 锟斤拷锟矫伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷印锟�
    • 使锟斤拷 ACE_agent 锟斤拷始锟斤拷强锟斤拷学习锟斤拷锟斤拷锟藉，锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷诤锟斤拷锟斤拷锟斤拷锟斤拷锟窖帮拷锟斤拷锟节伙拷锟斤拷锟斤拷锟叫讹拷锟斤拷
    • 锟斤拷锟斤拷锟芥储锟斤拷锟斤拷图锟斤拷锟斤拷目录锟斤拷确锟斤拷训锟斤拷锟斤拷锟斤拷锟叫碉拷锟斤拷锟矫猴拷锟斤拷锟饺拷鼗岜伙拷锟铰硷拷锟斤拷锟斤拷锟�
    • 锟斤拷始锟斤拷锟斤拷锟斤拷锟斤拷锟截放伙拷锟斤拷锟斤拷锟斤拷 memory 锟斤拷锟节存储锟斤拷锟叫碉拷锟斤拷史锟斤拷锟捷ｏ拷 local_buffer 锟斤拷锟斤拷锟斤拷锟斤拷锟饺拷氐母锟斤拷隆锟�

  2. 锟斤拷训锟斤拷循锟斤拷 :

    • 锟斤拷锟斤拷锟斤拷锟斤拷 锟斤拷锟斤拷锟斤拷懿锟斤拷锟斤拷锟叫★拷锟斤拷锟接伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟接诧拷锟斤拷锟斤拷选锟斤拷锟斤拷锟斤拷通锟斤拷锟斤拷锟街凤拷式锟斤拷确锟斤拷锟斤拷锟斤拷探锟斤拷锟酵猴拷锟斤拷锟斤拷锟矫★拷

    • 锟斤拷锟斤拷锟斤拷锟饺拷锟� 锟斤拷锟斤拷锟截讹拷锟斤拷锟斤拷冢锟斤拷泳植锟斤拷锟斤拷锟斤拷锟斤拷胁锟斤拷锟斤拷锟斤拷荩锟酵拷锟� get_sa2r_weight 锟斤拷锟斤拷使锟斤拷DirectLiNGAM锟姐法锟斤拷锟斤拷佣锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷权锟截★拷锟斤拷锟饺拷鼗锟斤拷锟轿拷锟斤拷锟斤拷锟较拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷呕锟斤拷锟斤拷浴锟�

    • 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟� 锟斤拷锟斤拷 memory 锟叫碉拷锟斤拷锟斤拷锟姐够锟斤拷时锟斤拷锟斤拷始通锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷Q锟斤拷锟斤拷筒锟斤拷锟斤拷锟斤拷纾癸拷眉锟斤拷锟斤拷锟斤拷锟斤拷锟饺拷锟斤拷锟斤拷锟斤拷锟斤拷锟绞э拷锟斤拷锟斤拷锟�

    • 锟斤拷录锟诫保锟斤拷模锟斤拷 锟斤拷每锟斤拷一锟斤拷锟侥诧拷锟斤拷锟斤拷锟姐法锟斤拷锟斤拷缘锟角帮拷锟斤拷缘锟斤拷锟斤拷埽锟斤拷锟铰硷拷锟斤拷冉辖锟斤拷锟斤拷欠癯锟斤拷锟绞凤拷锟斤拷值锟斤拷锟斤拷锟斤拷牵锟斤拷虮４锟侥ｏ拷偷募锟斤拷恪�

    • 使锟斤拷 wandb 锟斤拷录训锟斤拷锟斤拷锟斤拷锟叫碉拷指锟疥，锟斤拷锟斤拷锟斤拷失锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟饺拷氐募锟斤拷锟绞憋拷洌拷锟叫╋拷锟较拷锟斤拷园锟斤拷锟斤拷锟斤拷院头锟斤拷锟窖碉拷锟斤拷锟斤拷獭锟�

锟斤拷锟斤拷模锟斤拷锟斤拷爰笆碉拷锟�

锟斤拷锟斤拷锟斤拷锟侥ｏ拷锟�

锟斤拷锟斤拷锟斤拷锟侥ｏ拷锟� 锟斤拷要通锟斤拷 get_sa2r_weight 锟斤拷锟斤拷实锟街ｏ拷锟斤拷锟斤拷锟斤拷 DirectLiNGAM 模锟酵斤拷希锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟饺拷亍锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷训锟斤拷循锟斤拷锟斤拷锟斤拷锟铰ｏ拷 

causal_weight, causal_computing_time = get_sa2r_weight(env, local_buffer, agent, sample_size=config.causal_sample_size, causal_method='DirectLiNGAM')

锟斤拷锟斤拷锟斤拷锟斤拷锟轿ｏ拷 get_sa2r_weight 锟斤拷锟斤拷锟斤拷锟斤拷诘锟角帮拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷荩锟� local_buffer 锟斤拷锟斤拷锟斤拷锟侥ｏ拷停锟斤拷锟斤拷锟绞癸拷玫锟斤拷锟� DirectLiNGAM 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟叫讹拷锟斤拷氐锟斤拷锟斤拷权锟截ｏ拷 causal_weight 锟斤拷锟斤拷锟斤拷些权锟截伙拷影锟斤拷锟斤拷锟斤拷牟锟斤拷锟斤拷呕锟斤拷筒锟斤拷锟斤拷锟斤拷隆锟斤拷丶锟斤拷呒锟斤拷锟斤拷锟斤拷锟�

  1. 锟斤拷锟斤拷锟斤拷锟� 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟� total_numsteps % config.causal_sample_interval == 0 时锟斤拷锟斤拷锟斤拷确锟斤拷只锟斤拷指锟斤拷锟侥诧拷锟斤拷锟斤拷锟斤拷诩锟斤拷锟斤拷锟斤拷权锟截ｏ拷锟斤拷锟斤拷每一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷悖拷锟斤拷锟斤拷锟姐负锟斤拷锟斤拷
  2. 锟街诧拷锟斤拷锟斤拷锟斤拷 锟斤拷 local_buffer 锟叫存储锟斤拷锟姐够锟斤拷锟斤拷锟斤拷锟斤拷 config.causal_sample_size 锟斤拷锟斤拷锟斤拷些锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟较碉拷姆锟斤拷帧锟�
  3. 锟斤拷锟斤拷锟斤拷锟� 锟斤拷 DirectLiNGAM 锟斤拷选锟斤拷锟斤拷锟斤拷模锟酵ｏ拷锟斤拷锟节达拷状态锟斤拷锟叫讹拷锟酵斤拷锟斤拷之锟斤拷锟狡碉拷锟斤拷锟斤拷锟斤拷锟较碉拷锟�

锟斤拷锟饺拷丶锟斤拷锟斤拷锟缴后，筹拷锟斤拷峤拷锟叫┤拷锟接︼拷玫锟斤拷锟斤拷锟斤拷呕锟斤拷校锟斤拷锟斤拷壹锟铰既拷丶锟斤拷锟斤拷锟绞憋拷锟斤拷锟斤拷息锟斤拷 

def get_sa2r_weight(env, memory, agent, sample_size=5000, causal_method='DirectLiNGAM'):
    锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷
    return weight, model._running_time

锟斤拷锟斤拷锟斤拷锟侥猴拷锟斤拷锟斤拷锟斤拷锟斤拷DirectLiNGAM模锟酵硷拷锟斤拷锟斤拷锟阶刺拷锟斤拷锟斤拷锟斤拷徒锟斤拷锟街拷锟斤拷锟斤拷锟饺拷亍锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟絃aTeX锟斤拷式锟斤拷细锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟饺拷氐墓锟斤拷蹋锟�

  1. 锟斤拷锟斤拷预锟斤拷锟斤拷 锟斤拷
    锟斤拷锟斤拷 memory 锟叫诧拷锟斤拷锟斤拷 states 锟斤拷状态锟斤拷锟斤拷 actions 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 rewards 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷拼锟接ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟捷撅拷锟斤拷 \(X_{\text{ori}}\) 锟斤拷

    锟斤拷锟叫ｏ拷 \(S\) 锟斤拷锟斤拷状态锟斤拷 \(A\) 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 \(R\) 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟脚ｏ拷锟斤拷锟斤拷锟斤拷锟捷匡拷 \(X\) 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟�

  2. 锟斤拷锟侥ｏ拷锟斤拷锟斤拷 锟斤拷

    锟斤拷 X_ori 转锟斤拷为 X 锟斤拷为锟斤拷锟斤拷锟斤拷 pandas 锟斤拷锟捷匡拷谋锟斤拷锟斤拷院锟斤拷锟斤拷锟斤拷

    使锟斤拷 DirectLiNGAM 模锟酵对撅拷锟斤拷 \(X\) 锟斤拷锟斤拷锟斤拷希锟斤拷玫锟斤拷锟斤拷锟斤拷系锟斤拷锟节接撅拷锟斤拷 \(A_{\text{model}}\) 锟斤拷

    锟斤拷锟节接撅拷锟斤拷锟绞咀刺拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟街拷锟斤拷锟斤拷锟斤拷峁癸拷锟斤拷乇锟斤拷谴佣锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟接帮拷锟斤拷系锟斤拷

  3. 锟斤拷取锟斤拷锟斤拷锟皆斤拷锟斤拷锟斤拷锟斤拷锟饺拷锟� 锟斤拷
    通锟斤拷锟节接撅拷锟斤拷锟斤拷取锟斤拷锟斤拷锟皆斤拷锟斤拷锟斤拷锟斤拷锟饺拷锟� \(w_{\text{r}}\) 锟斤拷锟斤拷权锟截达拷锟节接撅拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷选锟斤拷锟诫动锟斤拷锟斤拷应锟斤拷元锟截ｏ拷

    锟斤拷锟叫ｏ拷 \(d_s\) 锟斤拷状态锟斤拷维锟饺ｏ拷 \(d_a\) 锟角讹拷锟斤拷锟斤拷维锟饺★拷

  4. 锟斤拷锟饺拷氐墓锟揭伙拷锟� 锟斤拷
    锟斤拷锟斤拷锟饺拷锟� \(w_{\text{r}}\) 锟斤拷锟斤拷Softmax锟斤拷一锟斤拷锟斤拷确锟斤拷锟斤拷锟角碉拷锟杰猴拷为1锟斤拷

  5. 锟斤拷锟斤拷权锟截的尺讹拷 锟斤拷
    锟斤拷锟斤拷锟斤拷权锟截革拷锟捷讹拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟脚ｏ拷

锟斤拷锟斤拷锟斤拷锟斤拷锟饺拷锟� \(w\) 锟斤拷示每锟斤拷锟斤拷锟斤拷锟皆斤拷锟斤拷锟斤拷锟斤拷锟接帮拷欤拷锟斤拷锟斤拷锟揭伙拷锟斤拷锟斤拷锟斤拷糯锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷诮锟揭伙拷锟斤拷牟锟斤拷缘锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷锟斤拷锟脚伙拷模锟斤拷

锟斤拷锟斤拷锟角对猴拷锟斤拷锟斤拷锟斤拷原锟斤拷锟斤拷锟金步斤拷锟酵ｏ拷

锟斤拷锟斤拷锟脚伙拷模锟斤拷 锟斤拷要锟斤拷 agent.update_parameters 锟斤拷锟斤拷实锟街★拷 agent.update_parameters 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟揭匡拷锟斤拷锟斤拷锟角匡拷锟窖帮拷懈锟斤拷虏锟斤拷锟� ( policy ) 锟酵硷拷值锟斤拷锟界（critic锟斤拷锟侥诧拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷堋锟斤拷锟斤拷锟斤拷锟斤拷实锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷员锟斤拷锟桔家ｏ拷SAC, Soft Actor-Critic锟斤拷锟侥革拷锟铰伙拷锟狡ｏ拷锟斤拷锟揭硷拷锟斤拷锟斤拷锟斤拷锟饺拷锟斤拷锟�"锟斤拷锟斤拷"锟斤拷元锟斤拷dormant neurons锟斤拷锟侥达拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷偷锟铰筹拷锟斤拷院锟斤拷榷锟斤拷浴锟� 

critic_1_loss, critic_2_loss, policy_loss, ent_loss, alpha, q_sac, dormant_metrics = agent.update_parameters(memory, causal_weight, config.batch_size, updates)

通锟斤拷 agent.update_parameters 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟铰硷拷锟斤拷锟斤拷锟街ｏ拷

  1. Critic锟斤拷锟界（锟斤拷值锟斤拷锟界） 锟斤拷 critic_1_loss 锟斤拷 critic_2_loss 锟街憋拷锟斤拷锟斤拷锟斤拷 Critic 锟斤拷锟斤拷锟斤拷锟绞э拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟角帮拷锟斤拷缘募锟街碉拷锟�
  2. Policy锟斤拷锟界（锟斤拷锟斤拷锟斤拷锟界） 锟斤拷 policy_loss 锟斤拷示锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟绞э拷锟斤拷锟斤拷锟斤拷呕锟� agent 锟斤拷锟叫讹拷选锟斤拷
  3. Entropy锟斤拷失 锟斤拷 ent_loss 锟斤拷锟斤拷锟斤拷锟节诧拷锟皆碉拷锟斤拷锟斤拷裕锟斤拷锟斤拷锟� agent 锟斤拷探锟斤拷锟斤拷锟斤拷锟斤拷之锟斤拷锟揭碉拷平锟解。
  4. Alpha 锟斤拷锟斤拷示锟斤拷锟斤拷应锟斤拷锟斤拷系锟斤拷锟斤拷锟斤拷锟节碉拷锟斤拷探锟斤拷锟斤拷锟斤拷锟斤拷之锟斤拷锟饺拷狻�

锟斤拷些锟斤拷锟斤拷锟侥革拷锟斤拷锟斤拷每锟斤拷训锟斤拷循锟斤拷锟叫憋拷锟斤拷锟矫ｏ拷锟斤拷使锟斤拷 wandb.log 锟斤拷录锟斤拷失锟斤拷锟斤拷锟斤拷锟斤拷氐锟窖碉拷锟斤拷锟斤拷荨锟�

update_parameters 锟斤拷 ACE_agent 锟斤拷锟叫碉拷一锟斤拷锟截硷拷锟斤拷锟斤拷锟斤拷锟斤拷锟节革拷锟捷撅拷锟斤拷胤呕锟斤拷锟斤拷锟斤拷械锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷偷牟锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷嵌锟斤拷涔わ拷锟皆拷锟斤拷锟斤拷锟较革拷锟斤拷停锟�

1. 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷锟饺ｏ拷锟斤拷锟斤拷锟斤拷 memory 锟叫诧拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷 state_batch 锟斤拷 action_batch 锟斤拷 reward_batch 锟斤拷 next_state_batch 锟斤拷 mask_batch 锟斤拷锟斤拷锟斤拷锟叫帮拷锟斤拷状态锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷状态锟皆硷拷锟斤拷锟诫，锟斤拷锟节憋拷示锟角凤拷为锟斤拷止状态锟斤拷 

state_batch, action_batch, reward_batch, next_state_batch, mask_batch = memory.sample(batch_size=batch_size)
  • state_batch 锟斤拷锟斤拷前锟斤拷状态锟斤拷
  • action_batch 锟斤拷锟节碉拷前状态锟斤拷执锟叫的讹拷锟斤拷锟斤拷
  • reward_batch 锟斤拷执锟叫该讹拷锟斤拷锟斤拷锟矫的斤拷锟斤拷锟斤拷
  • next_state_batch 锟斤拷执锟叫讹拷锟斤拷锟襟到达拷锟斤拷锟揭伙拷锟阶刺拷锟�
  • mask_batch 锟斤拷锟斤拷锟诫，锟斤拷锟节憋拷示锟角凤拷为锟斤拷止状态锟斤拷1 锟斤拷示锟斤拷锟斤拷止锟斤拷0 锟斤拷示锟斤拷止锟斤拷锟斤拷

2. 锟斤拷锟斤拷目锟斤拷 Q 值

锟斤拷锟矫碉拷前锟斤拷锟皆ｏ拷policy锟斤拷锟斤拷锟界，锟斤拷锟斤拷锟斤拷一锟斤拷状态锟侥讹拷锟斤拷 next_state_action 锟斤拷锟斤拷锟接︼拷母锟斤拷史植锟斤拷锟斤拷锟� next_state_log_pi 锟斤拷然锟斤拷锟斤拷锟斤拷目锟斤拷 Q 锟斤拷锟斤拷 critic_target 锟斤拷锟斤拷锟斤拷一时锟教碉拷锟斤拷小 Q 值锟斤拷锟斤拷锟斤拷辖锟斤拷锟斤拷锟斤拷劭锟斤拷锟斤拷锟� \(\gamma\) 锟斤拷锟斤拷锟斤拷一锟斤拷 Q 值锟斤拷 

with torch.no_grad():
    next_state_action, next_state_log_pi, _ = self.policy.sample(next_state_batch, causal_weight)
    qf1_next_target, qf2_next_target = self.critic_target(next_state_batch, next_state_action)
    min_qf_next_target = torch.min(qf1_next_target, qf2_next_target) - self.alpha * next_state_log_pi
    next_q_value = reward_batch + mask_batch * self.gamma * (min_qf_next_target)

  • 通锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 self.policy 为锟斤拷一锟斤拷状态 next_state_batch 锟斤拷锟斤拷锟斤拷锟斤拷 next_state_action 锟斤拷锟斤拷应锟侥诧拷锟斤拷锟斤拷 next_state_log_pi 锟斤拷

  • 使锟斤拷目锟斤拷 Q 锟斤拷锟斤拷锟斤拷锟� qf1_next_target 锟斤拷 qf2_next_target 锟斤拷锟斤拷取锟斤拷锟竭碉拷锟斤拷小值锟斤拷锟斤拷锟劫癸拷锟斤拷偏锟筋。

  • 锟斤拷锟斤拷使锟矫憋拷锟斤拷锟斤拷锟斤拷锟教硷拷锟斤拷 next_q_value 锟斤拷锟斤拷锟斤拷前锟侥斤拷锟斤拷锟斤拷锟斤拷锟桔匡拷锟斤拷锟斤拷 \(\gamma\) 锟斤拷锟斤拷锟斤拷一锟斤拷状态锟斤拷 Q 值锟斤拷

  • 锟斤拷锟斤， \(\alpha\) 锟斤拷锟斤拷锟斤拷锟饺拷兀锟斤拷锟斤拷锟狡斤拷锟教斤拷锟斤拷锟斤拷锟斤拷玫锟饺拷猓拷锟� mask_batch 锟斤拷为锟剿达拷锟斤拷锟斤拷止状态锟斤拷锟斤拷锟斤拷锟�

    使锟斤拷锟斤拷偏锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷目锟斤拷 Q 值锟斤拷通锟斤拷目锟斤拷锟斤拷锟斤拷 ( critic_target ) 锟斤拷锟斤拷锟斤拷锟揭伙拷锟阶刺拷投锟斤拷锟斤拷锟� Q 值锟斤拷锟斤拷使锟矫斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷碌锟角� Q 值

3. 锟斤拷锟斤拷 Q 锟斤拷锟斤拷

锟斤拷锟脚ｏ拷使锟矫碉拷前 Q 锟斤拷锟斤拷 critic 锟斤拷锟狡碉拷前状态锟酵讹拷锟斤拷锟铰碉拷 Q 值 \(Q_1\) 锟斤拷 \(Q_2\) 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷目锟斤拷 Q 值锟侥撅拷锟斤拷锟斤拷锟斤拷锟绞э拷锟�

锟斤拷锟斤拷 Q 锟斤拷锟斤拷锟斤拷锟斤拷锟绞э拷锟斤拷锟斤拷锟� Q 锟斤拷锟斤拷锟斤拷失之锟酵ｏ拷

然锟斤拷通锟斤拷锟斤拷锟津传诧拷 qf_loss 锟斤拷锟斤拷锟斤拷 Q 锟斤拷锟斤拷牟锟斤拷锟斤拷锟� 

qf1, qf2 = self.critic(state_batch, action_batch)
qf1_loss = F.mse_loss(qf1, next_q_value)
qf2_loss = F.mse_loss(qf2, next_q_value)
qf_loss = qf1_loss + qf2_loss

self.critic_optim.zero_grad()
qf_loss.backward()
self.critic_optim.step()
  • qf1 锟斤拷 qf2 锟斤拷锟斤拷锟斤拷 Q 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟节硷拷锟斤拷锟斤拷锟斤拷锟斤拷锟狡拷睢�
  • 锟斤拷失锟斤拷锟斤拷锟斤拷 Q 值锟侥撅拷锟斤拷锟斤拷睿∕SE锟斤拷锟斤拷 qf1_loss 锟斤拷 qf2_loss 锟街憋拷锟斤拷锟斤拷锟斤拷锟� Q 锟斤拷锟斤拷锟斤拷锟筋，锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷为锟杰碉拷 Q 锟斤拷失 qf_loss 锟斤拷
  • 通锟斤拷 self.critic_optim 锟脚伙拷锟斤拷锟斤拷锟斤拷失锟斤拷锟叫凤拷锟津传诧拷锟酵诧拷锟斤拷锟斤拷锟铰★拷

4. 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟�

每锟斤拷锟斤拷锟缴诧拷锟斤拷通锟斤拷 target_update_interval 锟斤拷锟狡ｏ拷锟斤拷锟斤拷始锟斤拷锟铰诧拷锟斤拷锟斤拷锟斤拷 policy 锟斤拷锟斤拷锟饺ｏ拷锟斤拷锟铰诧拷锟斤拷锟斤拷前状态锟铰的诧拷锟斤拷 \(\pi(a|s)\) 锟斤拷锟斤拷锟斤拷锟斤拷 Q 值锟斤拷锟斤拷权锟斤拷锟铰的诧拷锟斤拷锟斤拷失锟斤拷

锟斤拷锟斤拷锟绞拷锟斤拷锟斤拷虼锟斤拷锟斤拷虏锟斤拷锟斤拷锟斤拷纭� 

if updates % self.target_update_interval == 0:
    pi, log_pi, _ = self.policy.sample(state_batch, causal_weight)
    qf1_pi, qf2_pi = self.critic(state_batch, pi)
    min_qf_pi = torch.min(qf1_pi, qf2_pi)
    policy_loss = ((self.alpha * log_pi) - min_qf_pi).mean()

    self.policy_optim.zero_grad()
    policy_loss.backward()
    self.policy_optim.step()
  • 通锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷缘锟角白刺� state_batch 锟斤拷锟叫诧拷锟斤拷锟斤拷锟矫碉拷锟斤拷锟斤拷 pi 锟斤拷锟斤拷锟接︼拷牟锟斤拷锟斤拷锟� log_pi 锟斤拷
  • 锟斤拷锟斤拷锟斤拷锟斤拷锟绞� policy_loss 锟斤拷锟斤拷 \(\alpha\) 锟斤拷锟侥诧拷锟斤拷锟截硷拷去锟斤拷小锟斤拷 Q 值锟斤拷
  • 通锟斤拷 self.policy_optim 锟脚伙拷锟斤拷锟皆诧拷锟斤拷锟斤拷失锟斤拷锟叫凤拷锟津传诧拷锟酵诧拷锟斤拷锟斤拷锟铰★拷

5. 锟斤拷锟斤拷应锟截碉拷锟斤拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷远锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 automatic_entropy_tuning 锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 \(\alpha\) 锟斤拷锟斤拷失锟斤拷

锟斤拷通锟斤拷锟捷讹拷锟铰斤拷锟斤拷锟斤拷 \(\alpha\) 锟斤拷

锟斤拷锟� automatic_entropy_tuning 为锟芥，锟斤拷锟斤拷锟斤拷锟斤拷锟筋： 

if self.automatic_entropy_tuning:
    alpha_loss = -(self.log_alpha * (log_pi + self.target_entropy).detach()).mean()
    self.alpha_optim.zero_grad()
    alpha_loss.backward()
    self.alpha_optim.step()
    self.alpha = self.log_alpha.exp()
    alpha_tlogs = self.alpha.clone()
else:
    alpha_loss = torch.tensor(0.).to(self.device)
    alpha_tlogs = torch.tensor(self.alpha) # For TensorboardX logs
  • 通锟斤拷锟斤拷锟斤拷 alpha_loss 锟斤拷锟斤拷 self.alpha 锟斤拷锟斤拷锟斤拷锟斤拷锟皆碉拷探锟斤拷-锟斤拷锟斤拷平锟解。

6. 锟斤拷锟斤拷值

  • qf1_loss , qf2_loss : 锟斤拷锟斤拷 Q 锟斤拷锟斤拷锟斤拷锟绞�
  • policy_loss : 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟绞�
  • alpha_loss : 锟斤拷权锟截碉拷锟斤拷失
  • alpha_tlogs : 锟斤拷锟斤拷锟斤拷志锟斤拷录锟斤拷锟斤拷权锟斤拷
  • next_q_value : 平锟斤拷锟斤拷一锟斤拷 Q 值
  • dormant_metrics : 锟斤拷锟斤拷锟斤拷元锟斤拷锟斤拷囟锟斤拷锟�

锟斤拷锟矫伙拷锟斤拷模锟斤拷

锟斤拷锟矫伙拷锟斤拷模锟斤拷锟节达拷锟斤拷锟斤拷锟斤拷要锟斤拷锟斤拷锟斤拷 update_parameters 锟斤拷锟斤拷锟叫ｏ拷锟斤拷通锟斤拷 锟捷讹拷锟斤拷锟斤拷锟斤拷 (dominant metrics) 锟斤拷 锟脚讹拷锟斤拷锟斤拷 (perturbation functions) 实锟街对诧拷锟斤拷锟斤拷锟斤拷锟� Q 锟斤拷锟斤拷锟斤拷锟斤拷谩锟�

锟斤拷锟斤拷锟竭硷拷

锟斤拷锟斤拷锟斤拷锟斤拷锟借定锟斤拷 reset_interval 锟叫讹拷锟角凤拷锟斤拷要锟皆诧拷锟斤拷锟斤拷锟斤拷锟� Q 锟斤拷锟斤拷锟斤拷锟斤拷哦锟斤拷锟斤拷锟斤拷谩锟斤拷锟斤拷锟绞癸拷锟斤拷锟�"锟斤拷锟斤拷"锟斤拷元锟侥革拷锟筋，锟斤拷一些锟斤拷锟捷度革拷锟斤拷锟斤拷影锟斤拷锟叫★拷锟斤拷锟皆拷锟斤拷锟斤拷芑岜伙拷锟斤拷锟斤拷锟斤拷锟斤拷谩锟�

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟竭讹拷锟斤拷 dormant_metrics 锟斤拷锟斤拷锟饺拷夭锟斤拷锟� causal_diff 锟斤拷通锟斤拷锟脚讹拷锟斤拷锟斤拷 perturb_factor 锟斤拷锟斤拷锟斤拷锟角凤拷锟斤拷锟斤拷锟斤拷锟叫诧拷锟街伙拷全锟斤拷锟斤拷锟脚讹拷锟斤拷锟斤拷锟矫★拷

锟斤拷锟矫伙拷锟斤拷模锟斤拷锟皆拷锟�

锟斤拷锟矫伙拷锟斤拷锟斤拷要锟斤拷锟斤拷锟铰诧拷锟斤拷锟斤拷桑锟�

1. 锟斤拷锟斤拷锟捷讹拷锟斤拷锟斤拷锟斤拷 ( $\beta_\gamma $)

锟节革拷锟铰诧拷锟斤拷时锟斤拷锟斤拷锟斤拷 锟斤拷锟斤拷锟捷讹拷 锟斤拷锟斤拷某些锟截讹拷锟斤拷元锟斤拷锟斤拷锟斤拷诟锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷玫谋锟斤拷省锟斤拷锟斤拷锟斤拷锟酵拷锟斤拷锟斤拷锟� cal_dormant_grad(self.policy, type='policy', percentage=0.05) 实锟斤拷锟斤拷一锟斤拷锟姐，锟斤拷锟斤拷锟斤拷取锟斤拷 5% 锟斤拷锟斤拷锟斤拷锟捷讹拷锟斤拷锟斤拷为锟叫讹拷锟斤拷锟接★拷 

dormant_metrics = cal_dormant_grad(self.policy, type='policy', percentage=0.05)

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 ($ \beta_\gamma$ ) 锟斤拷权锟斤拷 ($ w$ )锟斤拷锟斤拷锟皆得碉拷锟斤拷锟叫в︼拷牟锟斤拷臁ｏ拷锟斤拷锟斤拷锟斤拷锟� causal_diff 锟斤拷锟斤拷示锟斤拷锟斤拷锟斤拷欤�

2. 锟斤拷锟斤拷锟矫诧拷锟皆猴拷 Q 锟斤拷锟斤拷

锟斤拷锟斤拷锟矫伙拷锟斤拷通锟斤拷平锟斤拷锟斤拷锟铰诧拷锟斤拷锟斤拷锟斤拷锟� Q 锟斤拷锟界，锟斤拷锟斤拷锟斤拷锟斤拷权锟截革拷锟铰碉拷锟铰碉拷锟斤拷锟界不锟饺讹拷锟斤拷锟斤拷锟节达拷锟斤拷锟斤拷锟斤拷 soft_update 实锟街ｏ拷 

soft_update(self.critic_target, self.critic, self.tau)

锟斤拷锟斤拷锟斤拷说锟斤拷锟斤拷锟斤拷锟铰的癸拷式为锟斤拷

锟斤拷锟叫ｏ拷( \(\tau\) ) 锟斤拷一锟斤拷锟斤拷小锟侥筹拷锟斤拷锟斤拷通锟斤拷锟斤拷锟斤拷 ( [0, 1] ) 之锟戒，确锟斤拷目锟斤拷锟斤拷锟斤拷母锟斤拷锟斤拷腔锟斤拷锟斤拷模锟斤拷锟斤拷锟斤拷学习锟斤拷锟饺讹拷锟皆★拷

3. 锟斤拷锟皆猴拷 Q 锟脚伙拷锟斤拷锟斤拷锟斤拷锟斤拷
4. 锟斤拷锟矫伙拷锟斤拷模锟斤拷锟接︼拷锟�

每锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟矫硷拷锟绞憋拷锟斤拷卸锟斤拷欠锟斤拷锟揭拷哦锟斤拷锟斤拷院锟� Q 锟斤拷锟界。通锟斤拷锟斤拷锟斤拷 perturb() 锟斤拷 dormant_perturb() 实锟街讹拷锟斤拷锟斤拷锟斤拷哦锟斤拷锟絧erturbation锟斤拷锟斤拷锟脚讹拷锟斤拷锟斤拷锟斤拷锟捷讹拷锟斤拷锟斤拷锟饺猴拷锟斤拷锟斤拷锟斤拷旃餐拷锟斤拷锟斤拷锟�

锟斤拷锟斤拷锟斤拷 Q 锟斤拷锟斤拷锟斤拷哦锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟铰凤拷锟斤拷锟斤拷

a. 锟斤拷锟矫硷拷锟斤拷锟斤拷时

锟斤拷锟斤拷锟斤拷每锟斤拷锟斤拷锟铰达拷锟斤拷 updates 锟斤到锟借定锟斤拷锟斤拷锟矫硷拷锟� self.reset_interval 锟斤拷锟斤拷锟斤拷 updates > 5000 时锟斤拷锟脚会触锟斤拷锟斤拷锟斤拷锟斤拷 Q 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷呒锟斤拷锟斤拷锟斤拷锟轿拷锟饺凤拷锟斤拷哦锟斤拷锟斤拷锟狡碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷诰锟斤拷锟揭伙拷谓铣锟斤拷锟窖碉拷锟绞憋拷锟斤拷锟斤拷小锟�

锟斤拷锟斤拷锟叫讹拷锟斤拷锟斤拷锟斤拷 

if updates % self.reset_interval == 0 and updates > 5000:
b. 锟斤拷锟斤拷锟捷度伙拷锟斤拷锟叫в︼拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟绞�

锟节达到锟斤拷锟斤拷锟矫硷拷锟斤拷锟斤拷锟斤拷然锟斤拷锟斤拷 锟捷讹拷锟斤拷锟斤拷锟斤拷 锟斤拷 锟斤拷锟叫в︼拷牟锟斤拷锟� 锟斤拷锟斤拷锟斤拷锟酵拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 causal_diff 锟斤拷锟捷讹拷锟斤拷锟斤拷锟斤拷 dormant_metrics['policy_grad_dormant_ratio'] 锟斤拷锟斤拷锟斤拷锟角凤拷锟斤拷要锟脚讹拷锟斤拷

  • 锟捷讹拷锟斤拷锟斤拷锟斤拷 锟斤拷锟姐方式通锟斤拷 cal_dormant_grad() 锟斤拷锟斤拷实锟街ｏ拷锟斤拷锟斤拷荻锟斤拷锟斤拷锟斤拷冉系停锟斤拷锟轿讹拷锟斤拷锟斤拷锟斤拷械锟侥承╋拷锟皆拷锟斤拷路锟斤拷裙锟叫★拷锟斤拷锟斤拷锟揭拷锟斤拷锟斤拷锟斤拷锟斤拷锟脚讹拷锟斤拷

  • 锟斤拷锟叫в︼拷锟斤拷锟� 通锟斤拷锟斤拷锟斤拷 causal_diff = np.max(causal_weight) - np.min(causal_weight) 锟矫碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟揭拷锟斤拷谩锟�

然锟斤拷锟斤拷锟斤拷锟叫┲低拷锟斤拷哦锟斤拷锟斤拷锟� factor 锟斤拷锟斤拷锟叫断ｏ拷 

factor = perturb_factor(dormant_metrics['policy_grad_dormant_ratio'])

锟斤拷锟斤拷哦锟斤拷锟斤拷锟� ( \(\text{factor} < 1\) )锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟脚讹拷锟斤拷 

if factor < 1:
    if self.reset == 'reset' or self.reset == 'causal_reset':
        perturb(self.policy, self.policy_optim, factor)
        perturb(self.critic, self.critic_optim, factor)
        perturb(self.critic_target, self.critic_optim, factor)
c. 锟杰斤拷
  • 锟斤拷锟铰达拷锟斤拷锟斤到锟借定锟斤拷锟斤拷锟矫硷拷锟� 锟斤拷锟揭撅拷锟斤拷锟斤拷一锟斤拷时锟斤拷锟窖碉拷锟斤拷锟� updates > 5000 锟斤拷锟斤拷
  • 锟捷讹拷锟斤拷锟斤拷锟斤拷 锟较低伙拷 锟斤拷锟叫в︼拷锟斤拷锟� 锟斤拷锟襟，碉拷锟铰硷拷锟斤拷锟斤拷锟斤拷哦锟斤拷锟斤拷锟� ( $\text{factor} < 1 $)锟斤拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷同时锟斤拷锟斤拷时锟斤拷锟斤拷锟皆猴拷 Q 锟斤拷锟界将锟斤拷锟脚讹拷锟斤拷锟斤拷锟矫★拷

锟脚讹拷锟斤拷锟接的硷拷锟斤拷

锟斤拷锟斤拷未锟斤拷锟斤拷校锟� factor 锟角伙拷锟斤拷锟斤拷锟斤拷锟斤拷锟捷讹拷锟斤拷锟斤拷锟饺伙拷锟斤拷锟斤拷锟叫в︼拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷哦锟斤拷锟斤拷印锟斤拷哦锟斤拷锟斤拷锟酵拷锟斤拷锟斤拷锟� perturb_factor() 锟斤拷锟叫硷拷锟姐，锟矫猴拷锟斤拷锟斤拷锟斤拷锟斤拷锟皆拷锟斤拷荻锟斤拷锟斤拷锟斤拷龋锟� dormant_ratio 锟斤拷锟斤拷锟斤拷锟叫в︼拷锟斤拷欤� causal_diff 锟斤拷锟斤拷锟斤拷锟斤拷 factor 锟侥达拷小锟斤拷

锟脚讹拷锟斤拷锟接ｏ拷factor锟斤拷

锟脚讹拷锟斤拷锟斤拷 factor 锟侥硷拷锟姐公式锟斤拷锟铰ｏ拷

锟斤拷锟叫ｏ拷

  • ( \(\text{dormant\_ratio}\) ) 锟斤拷锟斤拷锟斤拷锟斤拷锟捷讹拷锟斤拷锟斤拷锟饺ｏ拷锟斤拷锟斤拷示锟叫讹拷锟斤拷锟斤拷元锟斤拷锟捷度变化锟斤拷小锟斤拷锟斤拷锟竭接斤拷锟姐）锟斤拷锟斤拷锟节★拷锟斤拷锟竭★拷状态锟斤拷

  • ( \(\text{min\_perturb\_factor}\) ) 锟斤拷锟斤拷小锟脚讹拷锟斤拷锟斤拷值锟斤拷锟斤拷锟斤拷锟斤拷锟借定为 0.2 锟斤拷

  • ( \(\text{max\_perturb\_factor}\) ) 锟斤拷锟斤拷锟斤拷哦锟斤拷锟斤拷锟街碉拷锟斤拷锟斤拷锟斤拷锟斤拷瓒ㄎ� 0.9 锟斤拷

  • dormant_ratio :

    • 锟斤拷示锟斤拷锟斤拷锟叫达拷锟节★拷锟斤拷锟斤拷状态锟斤拷锟斤拷锟捷度憋拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟酵拷锟酵拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷荻确锟斤拷锟叫★拷锟侥筹拷锟斤拷锟街碉拷锟斤拷锟皆拷锟斤拷锟斤拷锟斤拷锟矫★拷 dormant_ratio 越锟襟，憋拷示越锟斤拷锟斤拷元锟斤拷锟捷度变化锟斤拷小锟斤拷说锟斤拷锟斤拷锟斤拷锟斤拷虏锟斤拷锟街ｏ拷锟斤拷要锟脚讹拷锟斤拷

  • max_perturb_factor :

    • 锟斤拷锟斤拷哦锟斤拷锟斤拷锟街碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷哦锟斤拷锟斤拷拥锟斤拷锟斤拷蓿锟斤拷锟斤拷锟斤拷锟斤拷瓒ㄎ� 0.9锟斤拷锟斤拷味锟斤拷锟斤拷锟斤拷哦锟斤拷锟斤拷炔锟斤拷岢拷锟� 90%锟斤拷

  • min_perturb_factor :

    • 锟斤拷小锟脚讹拷锟斤拷锟斤拷值锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟脚讹拷锟斤拷锟接碉拷锟斤拷锟睫ｏ拷锟斤拷锟斤拷锟斤拷锟借定为 0.2锟斤拷锟斤拷味锟脚硷拷使锟斤拷锟斤拷锟斤拷元锟斤拷锟斤拷锟杰低ｏ拷锟脚讹拷锟斤拷锟斤拷也锟斤拷锟斤拷小锟斤拷 20%锟斤拷

锟节硷拷锟斤拷锟斤拷锟叫в︼拷牟锟斤拷郑锟斤拷哦锟斤拷锟斤拷锟� factor 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷效应锟斤拷锟斤拷 causal_diff 锟斤拷锟斤拷锟斤拷锟斤拷 causal_diff 锟斤拷通锟斤拷锟斤拷锟斤拷锟斤拷锟叫в︼拷锟斤拷锟斤拷值锟斤拷锟斤拷小值锟侥诧拷锟斤拷锟斤拷锟斤拷玫模锟�

锟斤拷锟斤拷锟斤拷锟� causal_diff 锟斤拷影锟斤拷 causal_factor 锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷 factor 锟斤拷锟叫碉拷锟斤拷锟斤拷

锟斤拷锟斤拷哦锟斤拷锟斤拷拥墓锟绞�

锟斤拷锟斤拷锟斤拷选锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷茫锟� causal_reset 锟斤拷锟斤拷锟脚讹拷锟斤拷锟接斤拷使锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟� causal_factor 锟斤拷锟叫讹拷锟轿碉拷锟斤拷锟斤拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 factor 锟斤拷锟斤拷锟斤拷值锟斤拷锟斤拷锟捷讹拷锟斤拷锟斤拷锟饺伙拷锟斤拷锟叫в︼拷锟斤拷锟斤拷锟斤拷锟斤拷频模锟斤拷锟斤拷锟斤拷锟斤拷锟皆拷锟斤拷锟斤拷洗锟斤拷锟斤拷锟叫в︼拷锟斤拷锟较达拷时锟斤拷 factor 锟斤拷锟叫★拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟脚讹拷锟斤拷

锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷未锟斤拷锟斤拷锟揭碉拷锟斤拷锟斤拷锟角匡拷锟窖帮拷锟絉L锟斤拷训锟斤拷锟斤拷锟斤拷锟叫ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟藉（agent锟斤拷锟斤拷锟斤拷锟杰ｏ拷锟斤拷锟斤拷某些锟斤拷锟斤拷锟铰憋拷锟斤拷锟斤拷锟侥ｏ拷偷募锟斤拷恪ｏ拷锟斤拷强锟斤拷苑侄谓锟斤拷透么锟斤拷耄�

1. 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 

if i_episode % config.eval_interval == 0 and config.eval is True:

锟解部锟街达拷锟斤拷锟斤拷锟斤拷锟叫讹拷锟角凤拷应锟斤拷执锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟轿拷锟�

  • i_episode % config.eval_interval == 0 锟斤拷锟斤拷示每锟斤拷 config.eval_interval 锟斤拷训锟斤拷锟截合ｏ拷 i_episode 锟角碉拷前锟截猴拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷
  • config.eval is True 锟斤拷确锟斤拷 eval 锟斤拷锟斤拷为 True 锟斤拷也锟斤拷锟斤拷说锟斤拷锟斤拷锟斤拷锟斤拷锟杰匡拷锟斤拷锟斤拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷虢拷锟绞贾达拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟�

2. 锟斤拷始锟斤拷锟斤拷锟斤拷锟叫憋拷 

eval_reward_list = []

锟斤拷锟节存储每锟斤拷锟斤拷锟斤拷锟截合ｏ拷episode锟斤拷锟斤拷锟桔计斤拷锟斤拷锟斤拷锟皆憋拷之锟斤拷锟斤拷锟狡斤拷锟斤拷锟斤拷锟斤拷锟�

3. 锟斤拷锟斤拷锟斤拷锟斤拷 

for _ in range(config.eval_episodes):

锟斤拷锟斤拷锟阶段斤拷锟斤拷锟叫讹拷锟斤拷睾希锟斤拷锟� config.eval_episodes 指锟斤拷锟侥回猴拷锟斤拷锟斤拷锟斤拷锟皆伙拷锟斤拷锟斤拷锟斤拷锟侥憋拷锟街★拷

3.1 锟截合筹拷始锟斤拷 
state = env.reset()
episode_reward = []
done = False
  • env.reset() 锟斤拷锟斤拷锟矫伙拷锟斤拷锟斤拷锟斤拷贸锟绞甲刺� state 锟斤拷
  • episode_reward 锟斤拷锟斤拷始锟斤拷一锟斤拷锟叫憋拷锟斤拷锟斤拷锟节存储锟斤拷前锟截猴拷锟斤拷锟斤拷锟斤拷锟斤拷锟矫碉拷锟斤拷锟叫斤拷锟斤拷锟斤拷
  • done = False 锟斤拷锟斤拷 done 锟斤拷锟斤拷锟劫碉拷前锟截猴拷锟角凤拷锟斤拷锟斤拷锟�
3.2 执锟斤拷锟斤拷锟斤拷锟藉动锟斤拷 
while not done:
    action = agent.select_action(state, evaluate=True)
    next_state, reward, done, info = env.step(action)
    state = next_state
    episode_reward.append(reward)

  • 锟斤拷锟斤拷选锟斤拷 锟斤拷 agent.select_action(state, evaluate=True) 锟斤拷锟斤拷锟斤拷模式锟铰革拷锟捷碉拷前状态 state 选锟斤拷锟斤拷锟斤拷 evaluate=True 锟斤拷示锟斤拷选锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷模式锟铰ｏ拷通锟斤拷锟斤拷味锟斤拷探锟斤拷锟斤拷为锟斤拷锟截闭ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟教斤拷锟斤拷锟斤拷锟斤拷锟窖★拷锟斤拷锟斤拷哦锟斤拷锟斤拷锟斤拷锟�

  • 锟斤拷锟斤拷锟斤拷锟斤拷 锟斤拷 next_state, reward, done, info = env.step(action) 通锟斤拷执锟叫讹拷锟斤拷 action 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷状态 next_state 锟斤拷锟斤拷前锟斤拷锟斤拷 reward 锟斤拷锟截猴拷锟角凤拷锟斤拷锟斤拷谋锟街� done 锟斤拷锟皆硷拷锟斤拷锟斤拷锟斤拷息 info 锟斤拷

  • 状态锟斤拷锟斤拷 锟斤拷锟斤拷前状态锟斤拷锟斤拷锟斤拷为 next_state 锟斤拷锟斤拷锟斤拷锟斤拷玫慕锟斤拷锟� reward 锟芥储锟斤拷 episode_reward 锟叫憋拷锟叫★拷

循锟斤拷锟斤拷锟斤拷锟斤拷直锟斤拷锟截合斤拷锟斤拷锟斤拷锟斤拷 done == True 锟斤拷锟斤拷

3.3 锟芥储锟截合斤拷锟斤拷 
eval_reward_list.append(sum(episode_reward))

锟斤拷前锟截合斤拷锟斤拷锟斤拷锟桔计斤拷锟斤拷锟斤拷 sum(episode_reward) 锟斤拷锟斤拷锟斤拷锟接碉拷 eval_reward_list 锟斤拷锟斤拷锟节猴拷锟斤拷锟斤拷锟斤拷平锟斤拷锟斤拷锟斤拷锟斤拷

4. 锟斤拷锟斤拷平锟斤拷锟斤拷锟斤拷 

avg_reward = np.average(eval_reward_list)

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟截合斤拷锟斤拷锟襟，硷拷锟斤拷 eval_reward_list 锟斤拷平锟斤拷值 avg_reward 锟斤拷锟斤拷锟角碉拷前锟斤拷锟斤拷锟阶讹拷锟斤拷锟斤拷锟斤拷谋锟斤拷锟街革拷辍�

5. 锟斤拷锟斤拷锟斤拷锟侥ｏ拷锟� 

if config.save_checkpoint:
    if avg_reward >= best_reward:
        best_reward = avg_reward
        agent.save_checkpoint(checkpoint_path, 'best')
  • 锟斤拷锟� config.save_checkpoint 为 True 锟斤拷锟斤拷锟绞撅拷锟揭拷锟斤拷锟角否保达拷模锟酵★拷
  • 通锟斤拷锟叫讹拷 avg_reward 锟角否超癸拷锟斤拷之前锟斤拷锟斤拷呀锟斤拷锟� best_reward 锟斤拷锟斤拷锟斤拷牵锟斤拷锟斤拷锟斤拷 best_reward 锟斤拷锟斤拷锟斤拷锟芥当前模锟酵的硷拷锟姐。 
agent.save_checkpoint(checkpoint_path, 'best')

锟斤拷锟叫达拷锟斤拷峤拷锟斤拷锟斤拷锟斤拷状态锟斤拷锟芥到指锟斤拷锟斤拷路锟斤拷 checkpoint_path 锟斤拷锟斤拷锟斤拷锟轿� "best" 锟斤拷锟斤拷示锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷训锟侥ｏ拷汀锟�

锟斤拷锟侥革拷锟街斤拷锟�

锟饺咳ｏ拷锟斤拷锟皆凤拷锟街筹拷锟斤拷只锟斤拷录锟斤拷 0~1000 锟斤拷锟斤拷锟捷ｏ拷锟斤拷 1001 锟斤拷始锟斤拷每一锟斤拷锟斤拷锟捷讹拷锟斤拷示锟斤拷锟斤拷锟斤拷锟皆憋拷锟斤拷锟斤拷锟斤拷锟剿★拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷github锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟酵拷锟斤拷谋锟斤拷锟斤拷锟较�

锟斤拷锟斤拷锟斤拷息锟角ｏ拷锟斤拷锟斤拷 X+1 锟街达拷锟叫筹拷锟皆硷拷锟斤拷 X 锟街达拷锟叫碉拷锟斤拷息锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷荼锟斤拷锟斤拷锟斤拷锟斤拷锟�

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟缴ｏ拷锟斤拷锟皆硷拷也没锟斤拷 bug

锟斤拷锟斤拷锟斤拷锟斤拷锟侥匡拷锟斤拷锟斤拷耍锟斤拷锟揭拷锟斤拷锟斤拷锟斤拷锟斤拷目锟侥诧拷士师锟斤拷也锟斤拷业锟剿ｏ拷也锟斤拷锟斤拷说些什么锟斤拷锟斤拷然锟斤拷锟斤拷锟斤拷微锟脚ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥ｏ拷锟酵︼拷锟斤拷玫陌锟斤拷锟斤拷侄锟斤拷锟斤拷锟斤拷锟�

小锟斤拷锟狡硷拷锟侥讹拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥斤拷为锟斤拷锟斤拷锟斤拷息锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷同锟斤拷锟睫观碉拷锟街な碉拷锟斤拷锟斤拷锟斤拷锟�

a 1.0
a 1.0
锟斤拷锟酵ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷营状态锟斤拷锟斤拷式锟斤拷营锟斤拷锟斤拷锟斤拷锟皆ｏ拷锟斤拷锟斤拷 锟斤拷锟斤拷

锟斤拷戏锟斤拷锟斤拷

锟斤拷戏锟斤拷锟�

锟斤拷戏锟斤拷频

锟斤拷戏锟斤拷锟斤拷

锟斤拷戏锟筋动
锟斤拷alittletotheleft锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟筋备锟杰伙拷迎锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷戏锟斤拷锟斤拷业锟斤拷锟斤拷锟斤拷嵌锟斤拷粘锟斤拷锟斤拷锟斤拷械母锟斤拷锟斤拷锟斤拷锟斤拷锟�

锟斤拷锟斤拷锟斤拷锟斤拷

AWA 1.40
AWA 1.40
锟斤拷锟酵ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷营状态锟斤拷未知锟斤拷锟斤拷锟斤拷锟皆ｏ拷锟斤拷锟斤拷 锟斤拷锟斤拷

锟斤拷戏锟斤拷锟斤拷

锟斤拷戏锟斤拷锟�

锟斤拷戏锟斤拷频

锟斤拷戏锟斤拷锟斤拷

锟斤拷戏锟筋动
锟斤拷AWA锟斤拷锟斤拷卓锟斤拷锟斤拷锟缴匡拷锟斤拷锟斤拷MentalLab锟叫凤拷锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷色锟绞碉拷锟斤拷锟斤拷锟皆癸拷冒锟斤拷锟斤拷戏锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷戏锟斤拷锟芥，锟斤拷

锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷锟斤拷锟狡碉拷锟斤拷锟�

锟斤拷锟斤拷

同锟斤拷锟斤拷锟斤拷

锟斤拷锟斤拷

锟斤拷锟斤拷锟斤拷讯

锟斤拷锟斤拷

  • 锟斤拷戏锟狡硷拷
  • 锟斤拷匕姹�

锟斤拷锟斤拷
  • 锟斤拷讯
  • 锟斤拷频

锟斤拷锟斤拷
  • 锟斤拷卓锟饺帮拷
  • 苹锟斤拷锟饺帮拷

锟斤拷锟节猴拷锟斤拷锟斤拷锟斤拷系锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟�锟斤拷锟斤拷站锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟狡硷拷锟斤拷锟斤拷

频锟斤拷 锟斤拷锟斤拷频锟斤拷锟斤拷锟斤拷锟斤拷专锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷卓锟斤拷锟斤拷锟�锟斤拷app锟斤拷锟斤拷

锟狡硷拷 锟斤拷原锟斤拷锟斤拷锟斤拷锟斤拷锟铰帮拷锟斤拷锟斤拷锟斤拷锟杰匡拷锟斤拷锟斤拷锟截诧拷锟斤拷

锟斤拷锟斤拷 锟斤拷锟斤拷色锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 锟斤拷锟斤拷锟�锟斤拷锟斤拷战锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

扫锟斤拷维锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟街伙拷锟芥本锟斤拷

扫锟斤拷维锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷微锟脚癸拷锟节号ｏ拷

锟斤拷站锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟较达拷锟斤拷锟斤拷锟斤拷锟街凤拷锟斤拷陌锟饺拷锟斤拷敕拷始锟�[email protected]

锟斤拷ICP锟斤拷2022002427锟斤拷-10 锟芥公锟斤拷锟斤拷锟斤拷锟斤拷43070202000427锟斤拷© 2013~2025 haote.com 锟斤拷锟斤拷锟斤拷