锟斤拷锟斤拷位锟矫ｏ拷锟斤拷页 > 锟斤拷锟斤拷锟教筹拷 > 锟教筹拷 > Intel HDSLB 锟斤拷锟斤拷锟斤拷锟侥层负锟截撅拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷原锟斤拷锟酵诧拷锟斤拷锟斤拷锟斤拷

Intel HDSLB 锟斤拷锟斤拷锟斤拷锟侥层负锟截撅拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷原锟斤拷锟酵诧拷锟斤拷锟斤拷锟斤拷

锟斤拷源锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷|锟斤拷时锟戒：2024-05-27 08:46:07 |锟斤拷锟侥讹拷锟斤拷95锟斤拷|锟斤拷锟斤拷签锟斤拷 T 锟斤拷原 El S in 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 Intel 锟斤拷 |锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

锟斤拷篇锟斤拷要锟斤拷锟斤拷锟斤拷 Intel HDSLB 锟侥伙拷锟斤拷锟斤拷锟斤拷原锟斤拷锟酵诧拷锟斤拷锟斤拷锟矫的凤拷式锟斤拷希锟斤拷锟杰癸拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷顺锟斤拷锟侥帮拷 HDSLB-DPVS 锟斤拷目锟斤拷锟芥” 锟斤拷锟斤拷锟斤拷

前锟斤拷

锟斤拷锟斤拷一篇锟斤拷 Intel HDSLB 锟斤拷锟斤拷锟斤拷锟侥层负锟截撅拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟脚猴拷应锟矫筹拷锟斤拷锟斤拷锟叫ｏ拷锟斤拷锟斤拷锟斤拷锟截斤拷锟斤拷锟斤拷 HDSLB锟斤拷High Density Scalable Load Balancer锟斤拷锟斤拷锟杰度匡拷锟斤拷展锟侥革拷锟截撅拷锟斤拷锟斤拷锟斤拷锟斤拷为锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟侥层负锟截撅拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷位锟斤拷锟斤拷锟斤拷锟斤拷 HDSLB 锟斤拷锟狡硷拷锟斤拷捅锟皆碉拷锟斤拷锟接︼拷贸锟斤拷锟斤拷械锟斤拷锟斤拷锟斤拷锟斤拷疲锟斤拷约锟斤拷锟斤拷锟斤拷 HDSLB 锟斤拷锟斤拷锟杰诧拷锟斤拷锟斤拷锟捷★拷

锟劫斤拷一锟斤拷锟侥ｏ拷锟节憋拷篇锟斤拷锟斤拷锟斤拷锟斤拷要锟斤拷注 HDSLB 锟侥伙拷锟斤拷锟斤拷锟斤拷原锟斤拷锟酵诧拷锟斤拷锟斤拷锟矫凤拷式锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷实锟绞的诧拷锟斤拷锟斤拷为锟斤拷锟矫革拷锟姐泛锟侥匡拷锟斤拷锟斤拷锟角讹拷锟杰癸拷锟斤拷莘锟斤拷锟侥讹拷 HDSLB 展锟斤拷锟叫撅拷锟斤拷锟斤拷锟斤拷锟节憋拷篇锟叫伙拷锟斤拷锟� HDSLB-DPVS 锟斤拷源锟芥本锟斤拷锟斤拷锟叫斤拷锟杰★拷

HDSLB-DPVS 锟侥伙拷锟斤拷原锟斤拷

锟斤拷锟斤拷思锟藉，HDSLB-DPVS 锟角伙拷锟斤拷 DPVS 锟斤拷锟叫讹拷锟轿匡拷锟斤拷锟斤拷锟斤拷目锟斤拷锟斤拷 DPVS锟斤拷锟街筹拷为 DPDK-LVS锟斤拷锟斤拷一锟斤拷锟轿匡拷锟斤拷 LVS 锟节猴拷态锟侥层负锟截撅拷锟斤拷锟斤拷锟斤拷锟皆拷锟斤拷锟斤拷锟斤拷锟� DPDK 锟矫伙拷态锟斤拷锟斤拷锟斤拷锟斤拷倏锟杰斤拷锟叫匡拷锟斤拷锟斤拷锟侥层负锟截撅拷锟斤拷锟斤拷锟斤拷锟缴硷拷锟斤拷HDSLB-DPVS 锟侥硷拷锟斤拷锟斤拷栈锟斤拷要锟斤拷锟斤拷锟斤拷 4 锟斤拷锟斤拷锟斤拷锟斤拷桑锟�

LVS
DPDK
DPVS
HDSLB-DPVS

要锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 HDSLB-DPVS 锟侥伙拷锟斤拷实锟斤拷原锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷要锟斤拷头锟斤拷始锟斤拷锟斤拷

LVS

LVS锟斤拷Linux Virtual Server锟斤拷Linux 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟揭伙拷锟斤拷锟斤拷锟斤拷锟� 1998 锟斤拷锟斤拷牟愀猴拷鼐锟斤拷锟斤拷锟斤拷锟皆达拷锟侥匡拷锟斤拷锟侥匡拷锟斤拷锟绞癸拷锟� Local Balancer 锟斤拷锟斤拷锟斤拷 Server Cluster 锟斤拷锟斤拷锟斤拷实锟斤拷一锟斤拷锟斤拷锟斤拷锟斤拷锟矫匡拷锟斤拷锟斤拷锟皆ｏ拷Scalability锟斤拷锟斤拷锟缴匡拷锟皆ｏ拷Reliability锟斤拷锟酵可癸拷锟斤拷锟皆ｏ拷Manageability锟斤拷锟斤拷 Virtual Server锟斤拷

LVS锟斤拷 https://github.com/alibaba/LVS

Intel HDSLB 高性能四层负载均衡器 — 基本原理和部署配置

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷然 LVS 锟斤拷锟斤拷 Kernel 实锟街碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟窖撅拷锟斤拷锟斤拷时锟剿ｏ拷锟斤拷锟斤拷锟竭硷拷锟杰癸拷锟斤拷锟斤拷撇锟斤拷妫琇VS 锟侥猴拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟今，帮拷锟斤拷锟斤拷

VS锟斤拷Virtual Server锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟� 锟斤拷VS 锟斤拷锟斤拷 DS 锟斤拷 RS 锟斤拷瞎锟斤拷傻锟揭伙拷锟斤拷呒锟斤拷锟斤拷睿琕S 锟斤拷锟斤拷通锟斤拷一锟斤拷 VIP 锟斤拷锟解部 Clients 锟结供锟斤拷锟斤拷
DS锟斤拷Director Server锟斤拷锟斤拷锟斤拷锟斤拷锟饺凤拷锟斤拷锟斤拷锟斤拷 锟斤拷锟角充当 LB 锟斤拷锟斤拷锟斤拷诘姆锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷鼐锟斤拷锟斤拷锟皆碉拷执锟叫猴拷锟斤拷锟斤拷锟街凤拷锟斤拷锟斤拷锟斤拷也锟斤拷为 FE锟斤拷前锟剿凤拷锟斤拷锟斤拷锟斤拷锟斤拷
RS锟斤拷Real Server锟斤拷锟斤拷实锟斤拷锟斤拷锟斤拷锟斤拷 锟斤拷RS 锟斤拷锟斤拷锟斤拷锟斤拷锟节达拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥凤拷锟斤拷锟斤拷锟斤拷也锟斤拷为 BE锟斤拷锟斤拷朔锟斤拷锟斤拷锟斤拷锟斤拷锟�
VIP锟斤拷Virtual IP锟斤拷锟斤拷锟斤拷 IP 锟斤拷址锟斤拷 锟斤拷VS 锟斤拷锟解部 Client 锟结供锟斤拷锟斤拷锟� IP 锟斤拷址锟斤拷
DIP锟斤拷Director IP锟斤拷直锟斤拷 IP 锟斤拷址锟斤拷 锟斤拷Director Server 锟斤拷锟节诧拷锟斤拷 RS 锟斤拷锟斤拷通锟脚碉拷 IP 锟斤拷址锟斤拷
RIP锟斤拷Real IP锟斤拷锟斤拷实 IP 锟斤拷址锟斤拷 锟斤拷RS 锟斤拷 DS 锟斤拷通锟斤拷 IP 锟斤拷址锟斤拷
CIP锟斤拷Client IP锟斤拷锟酵伙拷锟剿碉拷 IP 锟斤拷址锟斤拷
NAT 锟斤拷锟斤拷锟斤拷锟阶拷锟侥Ｊ�
IP Tunneling 透锟斤拷转锟斤拷模式
DR 锟斤拷锟斤拷锟斤拷锟斤拷转锟斤拷模式
锟饺碉拷

Intel HDSLB 高性能四层负载均衡器 — 基本原理和部署配置

锟斤拷锟斤拷 LVS 锟斤拷锟斤拷细锟斤拷锟斤拷锟捷ｏ拷锟狡硷拷锟侥讹拷锟斤拷锟斤拷 LVS & Keepalived 实锟斤拷 L4 锟竭匡拷锟矫革拷锟截撅拷锟斤拷锟斤拷锟斤拷

DPDK

锟斤拷锟斤拷 2010 锟疥，IEEE 802.3 锟斤拷准委员锟结发锟斤拷锟斤拷 40GbE 锟斤拷 100GbE 802.3ba 锟斤拷太锟斤拷锟斤拷准锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷式锟斤拷锟斤拷锟斤拷 100G 时锟斤拷锟斤拷锟斤拷锟斤拷时锟斤拷Linux 锟节猴拷协锟斤拷栈锟斤拷锟斤拷锟界处锟斤拷锟斤拷锟杰撅拷一直锟斤拷锟斤拷锟斤拷战锟斤拷锟饺匡拷锟斤拷锟斤拷锟斤拷锟捷ｏ拷

CPU 锟斤拷锟斤拷 Main Memory 锟斤拷锟斤拷要锟斤拷时锟斤拷为 65 锟斤拷锟诫。
锟斤拷 NUMA node 锟斤拷 Main Memory 锟斤拷锟斤拷 Copy 锟斤拷锟斤拷要锟斤拷时锟斤拷为 40 锟斤拷锟诫。
CPU 锟斤拷锟斤拷一锟斤拷硬锟斤拷锟叫讹拷锟斤拷锟斤拷要锟斤拷时锟斤拷为 100 微锟诫。

锟斤拷实锟斤拷锟较ｏ拷100G 锟斤拷锟斤拷锟斤拷锟斤拷为 2 锟斤拷 PPS锟斤拷锟斤拷每锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷时锟戒不锟杰筹拷锟斤拷 50 锟斤拷锟诫。

锟缴硷拷锟斤拷锟斤拷锟斤拷 Kernel 锟斤拷锟斤拷锟斤拷锟斤拷锟窖撅拷锟竭碉拷锟剿拐点，DPDK 为锟剿讹拷锟斤拷锟斤拷锟斤拷通锟斤拷锟斤拷锟叫硷拷锟劫硷拷锟斤拷实锟斤拷锟斤拷 100G 锟斤拷锟斤拷转锟斤拷锟斤拷

使锟斤拷锟矫伙拷态协锟斤拷栈锟斤拷锟斤拷锟节猴拷协锟斤拷栈锟斤拷Kernel by-pass (user space implementation).
使锟斤拷锟斤拷训锟斤拷锟斤拷锟叫断ｏ拷Polling instead of interrupt.
使锟矫讹拷吮锟教达拷锟斤拷锟斤拷叱蹋锟絊hare-nothing, per-CPU for key data (lockless).
锟斤拷 CPU 锟斤拷锟斤拷通锟脚ｏ拷Lockless message for high performance IPC.
RX Steering and CPU affinity (avoid context switch).
Zero Copy (avoid packet copy and syscalls).
Batching TX/RX.
etc...

DPDK锟斤拷 https://github.com/DPDK/dpdk

Intel HDSLB 高性能四层负载均衡器 — 基本原理和部署配置

锟斤拷锟斤拷 DPDK 锟斤拷锟斤拷细锟斤拷锟斤拷锟捷ｏ拷锟狡硷拷锟侥讹拷锟斤拷锟斤拷 DPDK 锟斤拷锟斤拷锟捷硷拷锟劫凤拷锟斤拷锟侥猴拷锟斤拷思锟斤拷锟斤拷

DPVS

锟斤拷锟较ｏ拷锟斤拷锟斤拷 LVS 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷一锟斤拷 Linux Kernel Module锟斤拷ipvs锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟睫凤拷锟斤拷锟斤拷锟街达拷锟斤拷锟斤拷锟斤拷锟斤拷锟皆癸拷锟节癸拷司 iqiyi 锟斤拷锟斤拷 DPDK 锟斤拷锟斤拷锟斤拷 DPVS锟斤拷值锟斤拷一锟斤拷锟斤拷牵锟斤拷锟斤拷锟� DPVS 锟斤拷目锟缴癸拷锟节癸拷司锟斤拷源锟斤拷维锟斤拷锟斤拷锟斤拷锟斤拷锟戒开源锟斤拷锟斤拷锟斤拷锟斤拷锟侥匡拷锟斤拷锟斤拷也锟斤拷锟斤拷锟斤拷押谩锟�

DPVS锟斤拷 https://github.com/iqiyi/dpvs

Intel HDSLB 高性能四层负载均衡器 — 基本原理和部署配置

锟斤拷锟斤拷锟斤拷锟杰凤拷锟斤拷锟斤拷呕锟街拷猓拷诠锟斤拷懿锟斤拷妫珼PVS 也锟结供锟剿革拷锟结富锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷

L4 Load Balancer, including FNAT, DR, Tunnel, DNAT modes, etc.
SNAT mode for Internet access from internal network.
NAT64 forwarding in FNAT mode for quick IPv6 adaptation without application changes.
Different schedule algorithms like RR, WLC, WRR, MH(Maglev Hashing), Conhash(Consistent Hashing) etc.
User-space Lite IP stack (IPv4/IPv6, Routing, ARP, Neighbor, ICMP ...).
Support KNI, VLAN, Bonding, Tunneling for different IDC environment.
Security aspect, support TCP syn-proxy, Conn-Limit, black-list锟斤拷 white-list.
QoS: Traffic Control.

锟斤拷锟斤拷锟斤拷锟杰癸拷锟斤拷锟芥，DPVS 锟斤拷锟斤拷锟斤拷锟斤拷锟捷凤拷锟斤拷芄锟斤拷突锟斤拷锟� Keepalived 锟斤拷 Master-Backup 锟竭匡拷锟矫架癸拷锟斤拷

ipvsadm锟斤拷锟斤拷锟斤拷 VS锟斤拷RS 锟斤拷锟竭硷拷锟斤拷源锟斤拷锟斤拷墓锟斤拷锟斤拷锟�
dpip锟斤拷锟斤拷锟斤拷 IP锟斤拷Route 锟饺伙拷锟斤拷锟斤拷锟斤拷锟斤拷源锟侥癸拷锟斤拷锟斤拷
keepalived锟斤拷锟斤拷锟斤拷锟结供锟斤拷锟斤拷 VRRP 协锟斤拷锟斤拷锟斤拷锟斤拷呖锟斤拷谩锟�

Intel HDSLB 高性能四层负载均衡器 — 基本原理和部署配置

HDSLB-DPVS

HDSLB-DPVS 锟斤拷 DPVS 锟斤拷锟斤拷锟斤拷锟斤拷为锟斤拷锟斤拷锟杰革拷锟截撅拷锟斤拷锟斤拷锟斤拷锟斤拷么锟斤拷锟竭的憋拷锟斤拷锟斤拷锟斤拷锟斤拷什么锟截ｏ拷锟金案撅拷锟角革拷强锟斤拷锟斤拷锟斤拷埽锟�

HDSLB-DPVS 锟斤拷 https://github.com/intel/high-density-scalable-load-balancer/tree/main

Intel HDSLB 高性能四层负载均衡器 — 基本原理和部署配置

通锟斤拷锟侥ｏ拷锟斤拷锟角匡拷锟斤拷使锟斤拷 RBP锟斤拷Ratio of Bandwidth and Performance growth rate锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟劫比ｏ拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷俦锟斤拷锟� CPU 锟斤拷锟杰碉拷锟斤拷锟劫ｏ拷锟斤拷锟斤拷RBP=BW GR/Perf. GR锟斤拷

锟斤拷锟斤拷图锟斤拷示锟斤拷2010 锟斤拷前锟斤拷锟斤拷锟斤拷拇锟斤拷锟斤拷昊拷锟斤拷锟斤拷锟皆硷拷锟� 30%锟斤拷锟斤拷 2015 锟斤拷锟斤拷锟斤拷锟斤拷 35%锟斤拷然锟斤拷锟节斤拷锟斤拷锏� 45%锟斤拷锟斤拷锟接︼拷模锟紺PU 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 10 锟斤拷前锟斤拷 23%锟斤拷锟铰斤拷锟斤拷 12%锟斤拷锟斤拷锟节斤拷锟斤拷直锟接斤拷锟酵碉拷 3.5%锟斤拷锟斤拷锟斤拷 3 锟斤拷时锟斤拷锟斤拷冢锟絉BP 指锟斤拷锟� RBP锟斤拷1 锟斤拷锟斤拷锟斤拷I/O 压锟斤拷锟斤拷未锟斤拷锟街筹拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷 RBP锟斤拷3锟斤拷锟斤拷锟节斤拷锟疥超锟斤拷锟斤拷 RBP锟斤拷10锟斤拷

锟缴硷拷锟斤拷CPU 锟斤拷锟斤拷锟窖撅拷锟睫凤拷直锟斤拷应锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷佟锟斤拷锟轿э拷锟� CPU 锟斤拷锟叫达拷锟斤拷锟斤拷锟斤拷锟劫碉拷 DPDK 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟秸斤拷锟�

Intel HDSLB 高性能四层负载均衡器 — 基本原理和部署配置

锟截碉拷 DPVS 锟斤拷 HDSLB-DPVS 锟侥憋拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷狻ｏ拷锟斤拷锟斤拷锟斤拷锟狡诧拷锟芥，DPVS 锟斤拷目锟斤拷锟角伙拷锟斤拷 DPDK 锟斤拷锟绞碉拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥硷拷锟劫ｏ拷锟斤拷 HDSLB-DPVS 锟斤拷锟斤拷锟揭伙拷锟斤拷慕锟斤拷锟斤拷旨锟斤拷锟斤拷锟斤拷氲� CPU 锟斤拷 NIC 锟斤拷锟斤拷锟较碉拷硬锟斤拷平台之锟较ｏ拷锟斤拷实锟斤拷锟斤拷锟斤拷锟斤拷锟杰度★拷锟斤拷锟斤拷锟斤拷锟斤拷展锟斤拷锟斤拷 2 锟斤拷目锟疥：

锟斤拷锟杰讹拷 锟斤拷指锟斤拷锟角碉拷锟斤拷 HDSLB 锟节碉拷锟� TCP 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟截憋拷摺锟�
锟斤拷锟斤拷展 锟斤拷指锟斤拷锟斤拷锟斤拷锟斤拷锟杰匡拷锟斤拷锟斤拷锟斤拷 CPU Core 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷源锟斤拷锟斤拷锟斤拷锟斤拷锟接讹拷锟斤拷锟斤拷锟斤拷展锟斤拷

实锟斤拷锟斤拷锟芥，锟斤拷锟斤拷锟斤拷锟酵碉拷 Intel Xeon CPU锟斤拷e.g. 3rd & 4th generation锟斤拷锟斤拷 E810 100G NIC 硬锟斤拷平台锟较ｏ拷实锟斤拷锟剿ｏ拷

Concurrent Session : 100M level / Node
Throughput : > 8Mpps / Core @FNAT
TCP Session Est. Rate > 800K / Core
Linear growth

锟皆此ｏ拷锟斤拷锟斤拷锟节★拷 Intel HDSLB 锟斤拷锟斤拷锟斤拷锟侥层负锟截撅拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟脚猴拷应锟矫筹拷锟斤拷锟斤拷锟斤拷锟斤拷锟窖撅拷锟斤拷 HDSLB-DPVS 锟斤拷越前锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟捷斤拷锟斤拷锟剿凤拷锟斤拷锟斤拷锟斤拷锟斤不锟斤拷赘锟斤拷锟斤拷

HDSLB 锟侥诧拷锟斤拷锟斤拷锟斤拷

硬锟斤拷要锟斤拷

锟斤拷锟斤拷锟斤拷氲绞碉拷锟斤拷锟斤拷冢锟斤拷锟揭拷锟阶� HDSLB-DPVS 锟侥憋拷锟诫、锟斤拷锟斤拷锟斤拷锟斤拷谩锟轿拷私锟斤拷涂锟斤拷锟斤拷锟斤拷偶锟斤拷锟斤拷锟斤拷员锟斤拷锟斤拷锟揭癸拷锟斤拷丝锟斤拷锟斤拷锟斤拷锟斤拷偶锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷胁锟斤拷锟酵碉拷锟皆★拷

	锟斤拷锟斤拷锟斤拷锟皆伙拷锟斤拷锟斤拷锟狡硷拷	锟斤拷锟解开锟斤拷锟斤拷锟斤拷锟脚硷拷锟狡硷拷
CPU 锟杰癸拷	Intel Xeon CPU 锟侥达拷	支锟斤拷 AVX512 系锟斤拷指锟筋集锟斤拷 Intel CPU 锟酵号ｏ拷锟斤拷锟界：Skylake 锟斤拷
CPU 锟斤拷源	2NUMA锟斤拷锟截闭筹拷锟竭筹拷	1NUMA锟斤拷4C
Memory 锟斤拷源	128G	16G
NIC 锟酵猴拷	Intel E810 100G	VirtI/O 锟斤拷锟斤拷锟斤拷支锟街讹拷锟斤拷锟�

锟斤拷锟斤拷 CPU 锟斤拷息锟斤拷

$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 57 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              2root@l4lb:~# lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 57 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              2
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           106
Model name:                      Intel(R) Xeon(R) Platinum 8350C CPU @ 2.60GHz
Stepping:                        6
CPU MHz:                         2599.994
BogoMIPS:                        5199.98
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       96 KiB
L1i cache:                       64 KiB
L2 cache:                        2.5 MiB
L3 cache:                        48 MiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Vulnerable
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_ts
                                 c arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_tim
                                 er aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb ibrs_enhanced fsgsbase tsc_adjust bm
                                 i1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xge
                                 tbv1 xsaves arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid arch_capabilities

锟斤拷锟斤拷要锟斤拷

OS锟斤拷Ubuntu 20.04.3
Kernel锟斤拷5.4.0-110-generic
GCC锟斤拷9.4.0
DPDK锟斤拷20.08

值锟斤拷注锟斤拷锟斤拷牵锟経buntu /boot 锟斤拷锟斤拷要锟斤拷锟斤拷 2G锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷诤锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷狻ｏ拷慰锟斤拷锟斤拷茫锟� https://askubuntu.com/questions/1207958/error-24-write-error-cannot-write-compressed-block

锟斤拷锟斤拷 OS 锟斤拷息锟斤拷

# 锟斤拷锟斤拷系统
$ sudo apt-get update -y && sudo apt-get upgrade -y
# Dev
$ sudo apt-get install git vim wget patch unzip -y
# popt
$ sudo apt-get install libpopt-dev -y
# NUMA
$ sudo apt-get install libnuma-dev -y
$ sudo apt-get install numactl -y
# Pcap
$ sudo apt-get install libpcap-dev -y 
# SSL
$ sudo apt-get install libssl-dev -y


# Kernel 5.4.0-136
$ uname -r
5.4.0-136-generic

$ ll /boot/vmlinuz*
lrwxrwxrwx 1 root root       25 Dec 27  2022 /boot/vmlinuz -> vmlinuz-5.4.0-136-generic
-rw------- 1 root root 13660416 Aug 10  2022 /boot/vmlinuz-5.4.0-125-generic
-rw------- 1 root root 13668608 Nov 24  2022 /boot/vmlinuz-5.4.0-136-generic
-rw------- 1 root root 11657976 Apr 21  2020 /boot/vmlinuz-5.4.0-26-generic
lrwxrwxrwx 1 root root       25 Dec 27  2022 /boot/vmlinuz.old -> vmlinuz-5.4.0-125-generic

$ dpkg -l | egrep "linux-(signed|modules|image|headers)" | grep $(uname -r)
ii  linux-headers-5.4.0-136-generic       5.4.0-136.153                     amd64        Linux kernel headers for version 5.4.0 on 64 bit x86 SMP
ii  linux-image-5.4.0-136-generic         5.4.0-136.153                     amd64        Signed kernel image generic
ii  linux-modules-5.4.0-136-generic       5.4.0-136.153                     amd64        Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
ii  linux-modules-extra-5.4.0-136-generic 5.4.0-136.153                     amd64        Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP

# GCC 9.4.0
$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

# 锟侥硷拷锟斤拷锟斤拷锟斤拷
$ ulimit -n 655350
$ echo "ulimit -n 655350" >> /etc/rc.local
$ chmod a+x /etc/rc.local

锟斤拷锟诫安装 DPDK

DPDK 锟斤拷装锟斤拷锟斤拷锟斤拷锟较革拷锟斤拷荩锟斤拷萍锟斤拷亩锟斤拷锟斤拷锟� DPDK 锟斤拷锟斤拷装锟斤拷锟斤拷锟斤拷锟斤拷

$ cd /root/
$ git clone https://github.com/intel/high-density-scalable-load-balancer hdslb

$ wget http://fast.dpdk.org/rel/dpdk-20.08.tar.xz
$ tar vxf dpdk-20.08.tar.xz

# 锟津补讹拷
$ cp hdslb/patch/dpdk-20.08/*.patch dpdk-20.08/
$ cd dpdk-20.08/
$ patch -p 1 < 0002-support-large_memory.patch
$ patch -p 1 < 0003-net-i40e-ice-support-rx-markid-ofb.patch

# 锟斤拷锟斤拷
$ make config T=x86_64-native-linuxapp-gcc MAKE_PAUSE=n
$ make MAKE_PAUSE=n -j 4

锟斤拷锟诫安装 HDSLB-DPVS

HDSLB-DPVS 锟侥憋拷锟诫安装锟斤拷锟斤拷锟斤拷锟斤拷要锟斤拷锟斤拷锟斤拷锟斤拷 CPU 硬锟斤拷锟斤拷锟斤拷指锟筋，锟斤拷锟界：AVX2锟斤拷AVX512 锟饺等★拷要锟斤拷锟斤拷晒锟斤拷锟斤拷锟� 2 锟斤拷锟斤拷锟揭拷锟�

要锟斤拷 CPU 硬锟斤拷支锟街ｏ拷锟狡硷拷使锟斤拷 Intel Xeon 锟斤拷锟斤拷锟斤拷锟斤拷系锟叫ｏ拷锟斤拷锟界：Intel Xeon Gold锟斤拷
要锟斤拷 GCC 锟芥本支锟街ｏ拷锟狡硷拷锟斤拷锟矫版本锟较高碉拷 GCC锟斤拷锟斤拷锟界本锟斤拷锟叫碉拷 9.4.0锟斤拷

$ cd dpdk-20.08/
$ export RTE_SDK=$PWD

$ cd hdslb/
$ chmod +x tools/keepalived/configure

# 锟斤拷锟诫安装
$ make -j 4
$ make install

锟斤拷锟矫达拷页锟节达拷

锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟皆伙拷锟斤拷锟叫ｏ拷锟斤拷页锟节达拷应锟矫撅拷锟斤拷锟杰的革拷锟斤拷HDSLB 锟斤拷 LB connect pool 锟斤拷要锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷诖妫拷锟斤拷锟绞碉拷实锟斤拷锟斤拷芄锟斤拷锟斤拷直锟接癸拷系锟斤拷

$ mkdir /mnt/huge_1GB
$ mount -t hugetlbfs nodev /mnt/huge_1GB

$ vim /etc/fstab
nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0

$ # for NUMA machine
$ echo 15 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages

$ vim /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="${GRUB_CMDLINE_LINUX_DEFAULT} default_hugepagesz=1G hugepagesz=1G hugepages=15" 

$ sudo update-grub
$ init 6

$ cat /proc/meminfo | grep Huge
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:      13
HugePages_Free:       13
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
Hugetlb:        13631488 kB

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi        13Gi       2.0Gi       2.0Mi       408Mi       2.2Gi
Swap:            0B          0B          0B

锟斤拷锟斤拷锟斤拷锟斤拷

$ modprobe vfio-pci
$ modprobe vfio enable_unsafe_noiommu_mode=1 # https://stackoverflow.com/questions/75840973/dpdk20-11-3-cannot-bind-device-to-vfio-pci
$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

$ cd dpdk-20.08/
$ export RTE_SDK=$PWD
$ insmod ${RTE_SDK}/build/kmod/rte_kni.ko

$ ${RTE_SDK}/usertools/dpdk-devbind.py --status-dev net
Network devices using kernel driver
===================================
0000:01:00.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused=vfio-pci *Active*
0000:03:00.0 'Virtio network device 1000' if=eth1 drv=virtio-pci unused=vfio-pci
0000:04:00.0 'Virtio network device 1000' if=eth2 drv=virtio-pci unused=vfio-pci


$ ifconfig eth1 down # 0000:03:00.0
$ ifconfig eth2 down # 0000:04:00.0
$ ${RTE_SDK}/usertools/dpdk-devbind.py -b vfio-pci 0000:03:00.0 0000:04:00.0

$ ${RTE_SDK}/usertools/dpdk-devbind.py --status-dev net
Network devices using DPDK-compatible driver
============================================
0000:03:00.0 'Virtio network device 1000' drv=vfio-pci unused=
0000:04:00.0 'Virtio network device 1000' drv=vfio-pci unused=
Network devices using kernel driver
===================================
0000:01:00.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused=vfio-pci *Active*

锟斤拷锟斤拷 HDSLB-DPVS

$ cp conf/hdslb.conf.sample /etc/hdslb.conf

# 锟斤拷锟矫斤拷锟斤拷
$ cat /etc/hdslb.conf
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! This is hdslb default configuration file.
!
! The attribute "" denotes the configuration item at initialization stage. Item of
! this type is configured oneshoot and not reloadable. If invalid value configured in the
! file, hdslb would use its default value.
!
! Note that hdslb configuration file supports the following comment type:
!   * line comment: using '#" or '!'
!   * inline range comment: using '<' and '>', put comment in between
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

! global config
global_defs {
    log_level   DEBUG # 锟斤拷锟斤拷锟斤拷锟�
    ! log_file    /var/log/hdslb.log
    ! log_async_mode    on
}

! netif config
netif_defs {
     pktpool_size     1048575
     pktpool_cache    256
    # LAN Interface 锟斤拷锟斤拷
     device dpdk0 {
        rx {
            queue_number        3
            descriptor_number   1024
            ! rss                 all
        }
        tx {
            queue_number        3
            descriptor_number   1024
        }
        fdir {
            mode                perfect
            pballoc             64k
            status              matched
        }
        ! promisc_mode
        kni_name                dpdk0.kni
    }
    # WAN Interface 锟斤拷锟斤拷
     device dpdk1 {
        rx {
            queue_number        3
            descriptor_number   1024
            ! rss                 all
        }
        tx {
            queue_number        3
            descriptor_number   1024
        }
        fdir {
            mode                perfect
            pballoc             64k
            status              matched
        }
        ! promisc_mode
        kni_name                dpdk1.kni
    }

    !  bonding bond0 {
    !    mode        0
    !    slave       dpdk0
    !    slave       dpdk1
    !    primary     dpdk0
    !    kni_name    bond0.kni
    !}
}

! worker config (lcores)
worker_defs {
    # control plane CPU
     worker cpu0 {
        type    master
        cpu_id  0
    }
    # data plane CPU
    # dpdk0锟斤拷1 锟斤拷 2 锟斤拷 Port 锟斤拷同一锟斤拷锟秸凤拷锟斤拷锟叫癸拷锟斤拷同一锟斤拷 CPU
     worker cpu1 {
        type    slave
        cpu_id  1
        port    dpdk0 {
            rx_queue_ids     0
            tx_queue_ids     0
            ! isol_rx_cpu_ids  9
            ! isol_rxq_ring_sz 1048576
        }
        port    dpdk1 {
            rx_queue_ids     0
            tx_queue_ids     0
            ! isol_rx_cpu_ids  9
            ! isol_rxq_ring_sz 1048576
        }
    }
     worker cpu2 {
        type    slave
        cpu_id  2
        port    dpdk0 {
            rx_queue_ids     1
            tx_queue_ids     1
            ! isol_rx_cpu_ids  10
            ! isol_rxq_ring_sz 1048576
        }
        port    dpdk1 {
            rx_queue_ids     1
            tx_queue_ids     1
            ! isol_rx_cpu_ids  10
            ! isol_rxq_ring_sz 1048576
        }
    }
     worker cpu3 {
        type    slave
        cpu_id  3
        port    dpdk0 {
            rx_queue_ids     2
            tx_queue_ids     2
            ! isol_rx_cpu_ids  11
            ! isol_rxq_ring_sz 1048576
        }
        port    dpdk1 {
            rx_queue_ids     2
            tx_queue_ids     2
            ! isol_rx_cpu_ids  11
            ! isol_rxq_ring_sz 1048576
        }
    }

}

! timer config
timer_defs {
    # cpu job loops to schedule dpdk timer management
    schedule_interval    500
}

! hdslb neighbor config
neigh_defs {
     unres_queue_length  128
     timeout             60
}

! hdslb ipv4 config
ipv4_defs {
    forwarding                 off
     default_ttl         64
    fragment {
         bucket_number   4096
         bucket_entries  16
         max_entries     4096
         ttl             1
    }
}

! hdslb ipv6 config
ipv6_defs {
    disable                     off
    forwarding                  off
    route6 {
         method           hlist
        recycle_time            10
    }
}

! control plane config
ctrl_defs {
    lcore_msg {
         ring_size                4096
        sync_msg_timeout_us             30000000
        priority_level                  low
    }
    ipc_msg {
         unix_domain /var/run/hdslb_ctrl
    }
}

! ipvs config
ipvs_defs {
    conn {
         conn_pool_size       2097152
         conn_pool_cache      256
        conn_init_timeout           30
        ! expire_quiescent_template
        ! fast_xmit_close
        !  redirect           off
    }

    udp {
        ! defence_udp_drop
        uoa_mode        opp
        uoa_max_trail   3
        timeout {
            normal      300
            last        3
        }
    }

    tcp {
        ! defence_tcp_drop
        timeout {
            none        2
            established 90
            syn_sent    3
            syn_recv    30
            fin_wait    7
            time_wait   7
            close       3
            close_wait  7
            last_ack    7
            listen      120
            synack      30
            last        2
        }
        synproxy {
            synack_options {
                mss             1452
                ttl             63
                sack
                ! wscale
                ! timestamp
            }
            ! defer_rs_syn
            rs_syn_max_retry    3
            ack_storm_thresh    10
            max_ack_saved       3
            conn_reuse_state {
                close
                time_wait
                ! fin_wait
                ! close_wait
                ! last_ack
           }
        }
    }
}

! sa_pool config
sa_pool {
    pool_hash_size   16
}

锟斤拷锟斤拷 HDSLB-DPVS

$ cd hdslb/
$ ./bin/hdslb
current thread affinity is set to F
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL:   Invalid NUMA socket, default to 0
EAL: Probe PCI driver: net_virtio (1af4:1000) device: 0000:01:00.0 (socket 0)
EAL:   Invalid NUMA socket, default to 0
EAL: Probe PCI driver: net_virtio (1af4:1000) device: 0000:03:00.0 (socket 0)
EAL:   using IOMMU type 8 (No-IOMMU)
EAL: Ignore mapping IO port bar(0)
EAL:   Invalid NUMA socket, default to 0
EAL: Probe PCI driver: net_virtio (1af4:1000) device: 0000:04:00.0 (socket 0)
EAL: Ignore mapping IO port bar(0)
EAL: No legacy callbacks, legacy socket not created
DPVS: HDSLB version: , build on 2024.05.24.14:37:02
CFG_FILE: Opening configuration file '/etc/hdslb.conf'.
CFG_FILE: log_level = WARNING
NETIF: dpdk0:rx_queue_number = 3
NETIF: dpdk1:rx_queue_number = 3
NETIF: worker cpu1:dpdk0 rx_queue_id += 0
NETIF: worker cpu1:dpdk0 tx_queue_id += 0
NETIF: worker cpu1:dpdk1 rx_queue_id += 0
NETIF: worker cpu1:dpdk1 tx_queue_id += 0
NETIF: worker cpu2:dpdk0 rx_queue_id += 1
NETIF: worker cpu2:dpdk0 tx_queue_id += 1
NETIF: worker cpu2:dpdk1 rx_queue_id += 1
NETIF: worker cpu2:dpdk1 tx_queue_id += 1
NETIF: worker cpu3:dpdk0 rx_queue_id += 2
NETIF: worker cpu3:dpdk0 tx_queue_id += 2
NETIF: worker cpu3:dpdk1 rx_queue_id += 2
NETIF: worker cpu3:dpdk1 tx_queue_id += 2
Kni: kni_add_dev: fail to set mac FA:27:00:00:0A:02 for dpdk0.kni: Timer expired
Kni: kni_add_dev: fail to set mac FA:27:00:00:0B:F6 for dpdk1.kni: Timer expired

Intel HDSLB 高性能四层负载均衡器 — 基本原理和部署配置

HDSLB-DPVS 锟斤拷锟斤拷锟斤拷锟斤拷锟襟，匡拷锟皆匡拷锟斤拷 2 锟斤拷 DPDK Port 锟酵讹拷应锟斤拷 2 锟斤拷 KNI Interface锟斤拷锟斤拷锟斤拷 DPDK Port 锟斤拷锟斤拷 LB 锟斤拷锟斤拷锟斤拷转锟斤拷锟斤拷锟斤拷 KNI 锟斤拷锟斤拷锟斤拷 Keepalived HA 锟斤拷锟斤拷模式锟斤拷

$ cd hdslb/bin/
$ ./dpip link show
1: dpdk0: socket 0 mtu 1500 rx-queue 3 tx-queue 3
    UP 10000 Mbps half-duplex auto-nego
    addr FA:27:00:00:0A:02 OF_TX_IP_CSUM
2: dpdk1: socket 0 mtu 1500 rx-queue 3 tx-queue 3
    UP 10000 Mbps half-duplex auto-nego
    addr FA:27:00:00:0B:F6 OF_TX_IP_CSUM


$ ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether fa:27:00:00:00:c0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.4/25 brd 192.168.0.127 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f827:ff:fe00:c0/64 scope link
       valid_lft forever preferred_lft forever
71: dpdk0.kni:  mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether fa:27:00:00:0a:02 brd ff:ff:ff:ff:ff:ff
72: dpdk1.kni:  mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether fa:27:00:00:0b:f6 brd ff:ff:ff:ff:ff:ff

锟斤拷锟斤拷 HDSLB-DPVS Two-arm Full-NAT 模式

Intel HDSLB 高性能四层负载均衡器 — 基本原理和部署配置

HDSLB-DPVS

$ cd hdslb/bin

# add VIP to WAN interface
$ ./dpip addr add 10.0.0.100/32 dev dpdk1

# route for WAN/LAN access
$ ./dpip route add 10.0.0.0/16 dev dpdk1
$ ./dpip route add 192.168.100.0/24 dev dpdk0

# add routes for other network or default route if needed.
$ ./dpip route show
inet 10.0.0.100/32 via 0.0.0.0 src 0.0.0.0 dev dpdk1 mtu 1500 tos 0 scope host metric 0 proto auto
inet 192.168.100.0/24 via 0.0.0.0 src 0.0.0.0 dev dpdk0 mtu 1500 tos 0 scope link metric 0 proto auto
inet 10.0.0.0/16 via 0.0.0.0 src 0.0.0.0 dev dpdk1 mtu 1500 tos 0 scope link metric 0 proto auto

# add service  to forwarding, scheduling mode is RR.
$ ./ipvsadm -A -t 10.0.0.100:80 -s rr

# add two RS for service, forwarding mode is FNAT (-b)
$ ./ipvsadm -a -t 10.0.0.100:80 -r 192.168.100.2 -b
$ ./ipvsadm -a -t 10.0.0.100:80 -r 192.168.100.3 -b

# add at least one Local-IP (LIP) for FNAT on LAN interface
$ ./ipvsadm --add-laddr -z 192.168.100.200 -t 10.0.0.100:80 -F dpdk0

# Check
$  ./ipvsadm -Ln
IP Virtual Server version 0.0.0 (size=0)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.100:80 rr
  -> 192.168.100.2:80             FullNat 1      0          0
  -> 192.168.100.3:80             FullNat 1      0          0

Server

$ python -m SimpleHTTPServer 80

Client

$ curl 10.0.0.100

锟斤拷锟斤拷锟斤拷锟�

锟斤拷锟斤拷 1 锟斤拷hdslb/tools/keepalived/configure 没锟斤拷执锟斤拷权锟睫★拷

make[1]: Leaving directory '/root/hdslb/src'
make[1]: Entering directory '/root/hdslb/tools'
if [ ! -f keepalived/Makefile ]; then \
        cd keepalived && \
        ./configure && \
        cd -; \
fi
/bin/sh: 3: ./configure: Permission denied
make[1]: *** [Makefile:29: keepalived_conf] Error 126
make[1]: Leaving directory '/root/hdslb/tools'
make: *** [Makefile:33: all] Error 1

# 锟斤拷锟�
$ chmod +x /root/hdslb/tools/keepalived/configure

锟斤拷锟斤拷 2 锟斤拷缺锟斤拷锟斤拷锟斤拷锟侥硷拷

Cause: ports in DPDK RTE (2) != ports in dpvs.conf(0)

# 锟斤拷锟�
$ cp conf/hdslb.conf.sample /etc/hdslb.conf

锟斤拷锟斤拷 3 锟斤拷锟斤拷锟斤拷锟斤拷 2MB hugepage size 太小

Cause: Cannot init mbuf pool on socket 0

# 锟斤拷锟斤拷锟斤拷锟� hugepagesize 锟斤拷锟斤拷为 1G
# ref锟斤拷https://stackoverflow.com/questions/51630926/cannot-create-mbuf-pool-with-dpdk

锟斤拷锟斤拷 4 锟斤拷缺锟斤拷 rte_kni 模锟斤拷

Cause: add KNI port fail, exiting...

# 锟斤拷锟�
$ insmod ${RTE_SDK}/build/kmod/rte_kni.ko

锟斤拷锟斤拷 5 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷页锟节存不锟斤拷

Kni: kni_add_dev: fail to set mac FA:27:00:00:07:AA for dpdk0.kni: Timer expired
Kni: kni_add_dev: fail to set mac FA:27:00:00:00:E1 for dpdk1.kni: Timer expired
IPSET: ipset_init: lcore 3: nothing to do.
IPVS: dp_vs_conn_init: lcore 3: nothing to do.
IPVS: fail to init synproxy: no memory
Segmentation fault (core dumped)

# 锟斤拷锟斤拷锟斤拷锟斤拷莸锟� 15G锟斤拷

锟斤拷锟斤拷 6 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷支锟斤拷 HDSLB-DPVS 锟斤拷要锟斤拷 hardware offloads 锟斤拷锟杰★拷

Kni: kni_add_dev: fail to set mac FA:27:00:00:0A:02 for dpdk0.kni: Timer expired
Kni: kni_add_dev: fail to set mac FA:27:00:00:0B:F6 for dpdk1.kni: Timer expired
Ethdev port_id=0 requested Rx offloads 0x62f doesn't match Rx offloads capabilities 0xa1d in rte_eth_dev_configure()
NETIF: netif_port_start: fail to config dpdk0
EAL: Error - exiting with code: 1
  Cause: Start dpdk0 failed, skipping ...

# 锟斤拷锟斤拷锟斤拷薷锟� netif 模锟介，锟斤拷锟斤拷锟斤拷锟斤拷支锟街碉拷 offloads 锟斤拷锟杰★拷
static struct rte_eth_conf default_port_conf = {
    .rxmode = {
......
        .offloads = 0,
        //.offloads = DEV_RX_OFFLOAD_CHECKSUM | DEV_RX_OFFLOAD_VLAN,
    },
......
    .txmode = {
......
        .offloads = 0,
        //.offloads = DEV_TX_OFFLOAD_IPV4_CKSUM | DEV_TX_OFFLOAD_UDP_CKSUM | DEV_TX_OFFLOAD_TCP_CKSUM | DEV_TX_OFFLOAD_MBUF_FAST_FREE,
    },

NOTE锟斤拷锟斤拷锟斤拷 DPDK 锟斤拷锟侥碉拷锟斤拷offloads mask 锟斤拷每锟斤拷 bit 锟斤拷锟斤拷锟斤拷锟斤拷锟截讹拷锟斤拷卸锟截癸拷锟杰★拷锟斤拷锟斤拷 0-15bit 锟斤拷应锟斤拷 Features锟斤拷

DEV_RX_OFFLOAD_VLAN_STRIP
DEV_RX_OFFLOAD_IPV4_CKSUM
DEV_RX_OFFLOAD_UDP_CKSUM
DEV_RX_OFFLOAD_TCP_CKSUM
DEV_RX_OFFLOAD_TCP_LRO
DEV_RX_OFFLOAD_QINQ_STRIP
DEV_RX_OFFLOAD_OUTER_IPV4_CKSUM
DEV_RX_OFFLOAD_MACSEC_STRIP
DEV_RX_OFFLOAD_VLAN_FILTER
DEV_RX_OFFLOAD_VLAN_EXTEND
DEV_RX_OFFLOAD_SCATTER
DEV_RX_OFFLOAD_TIMESTAMP
DEV_RX_OFFLOAD_SECURITY
DEV_RX_OFFLOAD_KEEP_CRC
DEV_RX_OFFLOAD_SCTP_CKSUM
DEV_RX_OFFLOAD_OUTER_UDP_CKSUM

锟斤拷锟斤拷 7 锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟界不支锟斤拷 RSS 锟斤拷锟斤拷小锟絭alid value: 0x0 锟斤拷示锟斤拷前锟斤拷锟斤拷锟斤拷支锟斤拷锟轿猴拷 RSS 锟斤拷希锟斤拷锟斤拷锟斤拷


Kni: kni_add_dev: fail to set mac FA:27:00:00:0A:02 for dpdk0.kni: Timer expired
Kni: kni_add_dev: fail to set mac FA:27:00:00:0B:F6 for dpdk1.kni: Timer expired
Ethdev port_id=0 invalid rss_hf: 0x3afbc, valid value: 0x0
NETIF: netif_port_start: fail to config dpdk0
EAL: Error - exiting with code: 1
  Cause: Start dpdk0 failed, skipping ...

# 锟斤拷锟斤拷锟绞� 1锟斤拷使锟斤拷支锟斤拷 multi-queues 锟斤拷 RSS hash 锟斤拷锟斤拷锟斤拷锟斤拷
# 锟斤拷锟斤拷锟绞� 2锟斤拷锟睫革拷 netif 模锟介，锟斤拷锟斤拷锟斤拷 multi-queues 锟斤拷 RSS hash 锟斤拷锟杰★拷
static struct rte_eth_conf default_port_conf = {
    .rxmode = {
        //.mq_mode        = ETH_MQ_RX_RSS,
        .mq_mode        = ETH_MQ_RX_NONE,
......
    },
    .rx_adv_conf = {
        .rss_conf = {
            .rss_key = NULL,
            //.rss_hf  = /*ETH_RSS_IP*/ ETH_RSS_TCP,
            .rss_hf  = 0
        },
    },
......
port->dev_conf.rx_adv_conf.rss_conf.rss_hf = 0;

锟斤拷锟斤拷 8 锟斤拷锟斤拷支锟街多播锟斤拷址锟斤拷锟斤拷

Kni: kni_add_dev: fail to set mac FA:27:00:00:0A:02 for dpdk0.kni: Timer expired
Kni: kni_add_dev: fail to set mac FA:27:00:00:0B:F6 for dpdk1.kni: Timer expired
NETIF: multicast address add failed for device dpdk0
EAL: Error - exiting with code: 1
  Cause: Start dpdk0 failed, skipping ...

# 锟斤拷锟斤拷锟斤拷乇斩嗖ワ拷锟斤拷锟�
    //ret = idev_add_mcast_init(port);
    //if (ret != EDPVS_OK) {
    //    RTE_LOG(WARNING, NETIF, "multicast address add failed for device %s\n", port->name);
    //    return ret;
    //}

锟斤拷锟斤拷 9 锟斤拷LB connect pool 锟节达拷太小锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷顺锟斤拷锟�

$ ./ipvsadm -A -t 10.0.0.100:80 -s rr
[sockopt_msg_recv] socket msg header recv error -- 0/88 recieved  

IPVS: lb_conn_hash_table_init: lcore 0: create conn_hash_tbl failed. Requested size: 1073741824 bytes. LB_CONN_CACHE_LINES_DEF: 1, LB_CONN_TBL_SIZE: 16777216

# 锟斤拷锟斤拷锟绞� 1锟斤拷锟斤拷锟斤拷锟接达拷页锟节存到实锟斤拷锟斤拷要锟侥达拷小锟斤拷
# 锟斤拷锟斤拷锟绞� 2锟斤拷
#	1锟斤拷锟斤拷锟酵凤拷一锟斤拷 lcore 锟侥达拷页锟节达拷
#	2锟斤拷锟斤拷锟斤拷小 DPVS_CONN_POOL_SIZE_DEF 锟斤拷 2097152 锟斤拷锟劫碉拷 1048576
//#define DPVS_CONN_POOL_SIZE_DEF     2097152
#define DPVS_CONN_POOL_SIZE_DEF     1048576

锟斤拷锟斤拷 10 锟斤拷锟斤拷锟斤拷锟斤拷锟芥本锟斤拷缺锟劫憋拷锟斤拷指锟筋。

error: inlining failed in call to always_inline   "'_mm256_cmpeq_epi64_mask':"  : target specific option mismatch

# 锟斤拷锟斤拷锟�
# 1锟斤拷锟斤拷锟斤拷 GCC 锟芥本锟斤拷 9.4.0
# 2锟斤拷确锟斤拷 CPU 支锟斤拷指锟筋集锟斤拷ref锟斤拷https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#expand=3828,301,2553&text=_mm256_cmpeq_epi64_mask&ig_expand=872

Intel HDSLB 高性能四层负载均衡器 — 基本原理和部署配置

锟斤拷锟�

值锟斤拷注锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷录锟角憋拷锟斤拷锟节碉拷锟戒开锟斤拷锟斤拷锟叫碉拷锟皆筹拷锟斤拷时锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟解，实锟斤拷锟斤拷锟斤拷一锟斤拷锟斤拷源锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷曰锟斤拷锟酵拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷源锟斤拷锟姐导锟铰的大部凤拷锟斤拷锟解。

锟斤拷螅锟狡拷锟揭拷锟斤拷锟斤拷锟� Intel HDSLB 锟侥伙拷锟斤拷锟斤拷锟斤拷原锟斤拷锟酵诧拷锟斤拷锟斤拷锟矫的凤拷式锟斤拷希锟斤拷锟杰癸拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷顺锟斤拷锟侥帮拷 HDSLB-DPVS 锟斤拷目锟斤拷锟芥” 锟斤拷锟斤拷锟斤拷锟斤拷锟芥，锟斤拷锟角斤拷锟劫次匡拷锟斤拷锟斤拷锟斤拷锟斤拷锟侥伙拷锟斤拷之锟较ｏ拷通锟斤拷锟斤拷Intel HDSLB 锟斤拷锟斤拷锟斤拷锟侥层负锟截撅拷锟斤拷锟斤拷锟斤拷锟竭硷拷锟斤拷锟皆和达拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟节撅拷 HDSLB-DPVS 锟侥高硷拷锟斤拷锟皆★拷锟斤拷锟斤拷锟杰癸拷锟斤拷锟斤拷锟酵达拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷锟斤拷诖锟斤拷锟斤拷锟斤拷锟�

小锟斤拷锟狡硷拷锟侥讹拷