啟動SR-IOV
===
###### tags: `K8s` `CNI` `PCI-e`
# 事前準備
* 先看網卡型號
```shell=
$ lspci | grep -i ethernet
```
前兩張是沒有支援SR-IOV 而後面8張是有支援SR-IOV網卡
* 選擇要確認的網卡
# Install SR-IOV to server
* 首先我們需要有SR-IOV driver然後我們要使用intel i40e driver去enable SR-IOV，但也請選擇適合你網卡的driver，我用Intel Corporation I350 Gigabit Network Connection為例
### Install i40e driver
```shell=
sudo apt install -y make gcc libelf-dev
I40E_VER=2.4.10
wget https://downloadmirror.intel.com/28306/eng/i40e-${I40E_VER}.tar.gz && \
tar xvzf i40e-${I40E_VER}.tar.gz && cd i40e-${I40E_VER}/src && sudo make install && cd -
```
### Update GRUB Settings
```shell=
sudo sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/c\GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"' /etc/default/grub
sudo sed -i '/GRUB_CMDLINE_LINUX/c\GRUB_CMDLINE_LINUX="intel_iommu=on"' /etc/default/grub
sudo update-grub
```
### Setup vfio-pci module auto-load on boot
```shell=
echo 'vfio-pci' | sudo tee /etc/modules-load.d/vfio-pci.conf
wget -qO- https://fast.dpdk.org/rel/dpdk-17.11.2.tar.xz | sudo tar -xJC /opt
sudo mv /opt/dpdk-* /opt/dpdk
```
### Create SR-IOV Script for systemctl
* 我們會寫一source code到/opt/scripts/sriov.sh我們需要佈署server 就要執行
```shell=
sudo mkdir -p /sriov-cni /opt/scripts
sudo su
cat << "EOF" > /opt/scripts/sriov.sh
#!/bin/bash
# Copied from infra/sriov.sh
# Usage: ./sriov.sh ens785f0
NUM_VFS=$(cat /sys/class/net/$1/device/sriov_totalvfs)
echo 0 | sudo tee /sys/class/net/$1/device/sriov_numvfs
echo $NUM_VFS | sudo tee /sys/class/net/$1/device/sriov_numvfs
sudo ip link set $1 up
for ((i = 0 ; i < ${NUM_VFS} ; i++ )); do ip link set $1 vf $i spoofchk off; done
for ((i = 0 ; i < ${NUM_VFS} ; i++ )); do ip link set dev $1 vf $i state enable; done
EOF
exit
# Script perms
sudo chmod 744 /opt/scripts/sriov.sh
```
* After we have the script, we can write a sriov.service to define a service and control this service by sudo systemctl enable sriov.
```shell=
sudo su
# Systemd unit to run the above script
cat << "EOF" > /etc/systemd/system/sriov.service
[Unit]
Description=Create VFs for ens802f0
[Service]
Type=oneshot
ExecStart=/opt/scripts/sriov.sh ens802f0
[Install]
WantedBy=default.target
EOF
exit
# Enable the SRIOV systemd unit
sudo systemctl enable sriov
```
# Concept of building SR-IOV supported Kubernetes
## Repositories to use
* 我們需要SR-IOV Container Network Interface 去建sriov binary 而SR-IOV Network Device Plugin 會去建sriovdp binary
* [sriov-cni](https://github.com/intel/sriov-cni)
* [sriov-network-device-plugin](https://github.com/intel/sriov-network-device-plugin)
### SR-IOV CNI
* SR-IOV CNI has simple work to do, like as:
* VF network plumbing
* VF allocation to POD network namespace
* VF deallocation from POD network namespace
* SR-IOV會用PF名稱去選擇VF還有配置VF給POD interface:
```yaml=
{
"name": "mynet",
"type": "sriov",
"master": "enp1s0f1",
"ipam": {
"type": "host-local",
"subnet": "10.55.206.0/26",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "10.55.206.1"
}
}
```
### SR-IOV Network Device Plugin
* 當我們需要配置VF到container，如果我們沒有足夠的VF可以用，由於SR-IOV CNI耗盡將不能被排程，所以SR-IOV network device plugin需要去支援:
1. 抓到在node上的SR-IOV Network interface
2. 監控VF的健康
3. 要能在kubernetes cluster上有資源佈署的限制
4. 能創建POD特別的network配置給配置完的VF
* SR-IOV network device plugin 使用 `/etc/pcidp/config.json` 去確定是哪一個PCI address 接著去run GO來配置SR-IOV給container
### Illustration of SR-IOV and SR-IOV Network Device Plugin
* 下面figure.1會大略解釋kubelet會如何與Multus與SR-IOV CNI&network device plugin來合作:
* 故事是這樣:
* 當我們要啟動SR-IOV network plugin 在每一個節點上當作一個DaemonSet，他會註冊`intel.com/sriov`當作kubernetes上的資源，Kubernetes會通過SR-IOV network plugin查看在各節點上的SR-IOV資源
* 當我們要創建一個基於Deployment的服務kubernetes會決定去配置一個有剩餘資源的節點上佈署
* 有節點提供Pod佈署，kuberntes會送network setup的請求給Mutlus，接著Multus會告訴SR-IOV CNI有關`POD od`還有`network configutation`。
* SR-IOV CNI會setup network 給VF 還有配置VF給Container當作interface。
* figure
## Deploying the cluster
* SR-IOV enabled on the network interface.
```shell=
cord@node1:~$ ip link show dev ens802f0
4: ens802f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether 00:1e:67:d2:ee:ea brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking off, link-state auto, trust off, query_rss off
vf 1 MAC 16:17:47:f9:43:9a, spoof checking off, link-state auto, trust off, query_rss off
vf 2 MAC fe:96:60:5f:d3:50, spoof checking off, link-state auto, trust off, query_rss off
vf 3 MAC 36:91:06:87:0d:c6, spoof checking off, link-state auto, trust off, query_rss off
... snip
```
* SR-IOV CNI located on CNI binary folder`
```shell=
cord@node2:~$ ls /opt/cni/bin
bridge centralip cnishim dhcp flannel host-local ipvlan loopback macvlan multus portmap ptp sample sriov tuning vlan
```
* SR-IOV Network Device Plugin runs
```shell=
$ kubectl -n kube-system logs sriov-device-plugin-5qfxc
I0107 23:19:30.469685 13614 server.go:132] ListAndWatch(sriov): send updated devices &ListAndWatchResponse{Devices:[&Device{ID:0000:04:10.0,Health:Healthy,} &Device{ID:0000:04:10.2,Health:Healthy,} &Device{ID:0000:04:10.4,Health:Healthy,} &Device{ID:0000:04:10.6,Health:Healthy,} ... snip],}
```
## Check resource exists and discovers by Kubernetes
```shell=
$ kubectl get node node2 -o json | jq '.status.allocatable'
{
"cpu": "40",
"ephemeral-storage": "452695013856",
"hugepages-1Gi": "32Gi",
"intel.com/sriov": "63",
"memory": "32210384Ki",
"pods": "110"
}
```
# Conclusion
* 本篇主要是描述kubernetes在SR-IOV CNI and SR-IOV Network Device Plugin 並不是聚焦在實作只有闡明安裝步驟。