Kubeadm === ###### tags: `contaner platform` `DevOps` # Install with kubeadm ## 事前準備 * 下載docker ```shell= curl -fsSL https://get.docker.com/ | sh ``` * 我們可以安裝自己想要的版本已對應kubernetes需求的版本-[Docker-ce](https://hackmd.mcl.math.ncu.edu.tw/s/H1hjWutiV) * 我在安裝時要求版本是18.06但最新版是18.09會有問題 * 所有節點網路可以溝通。 * 所有節點需要設定 APT Docker Repository： ```shell= $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - $ sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" ``` * 所有節點需要確定設定 APT 與 YUM Kubernetes Repository： ```shell= $ curl -s "https://packages.cloud.google.com/apt/doc/apt-key.gpg" | sudo apt-key add - $ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list ```  * Kubernetes v1.8+ 要求關閉系統 Swap，若不關閉則需要修改 kubelet 設定參數，這邊可以利用以下指令關閉：[原因](https://serverfault.com/questions/881517/why-disable-swap-on-kubernetes) ```shell= $ swapoff -a && sysctl -w vm.swappiness=0 # 不同機器有差異 $ sed '/swap.img/d' -i /etc/fstab #記得/etc/fstab也要註解掉SWAP掛載。 ```  # Kubernetes Master 建立: * 首先更新 APT 來源，並且安裝 Kubernetes 元件與工具： ```shell= export KUBE_VERSION="1.13.4" $ sudo apt-get update && sudo apt-get install -y kubelet=${KUBE_VERSION}-00 kubeadm=${KUBE_VERSION}-00 kubectl=${KUBE_VERSION}-00 kubernetes-cni=0.6.0-00 ``` * 進行初始化 Master，這邊需要進入root使用者執行以下指令： ```shell= $ sudo su - $ kubeadm token generate 7vgghl.rrgczyb24uqqgj0z $ kubeadm init --service-cidr 172.16.0.0/12 \ --kubernetes-version v${KUBE_VERSION} \ --pod-network-cidr 172.31.0.0/16 \ --token 7vgghl.rrgczyb24uqqgj0z \ --apiserver-advertise-address 10.100.37.100 --- #如果失敗重來 #最前面加一條 --ignore-preflight-errors=all $ kubeadm init --ignore-preflight-errors=all \ --service-cidr 172.16.0.0/12 \ --kubernetes-version v${KUBE_VERSION} \ --pod-network-cidr 172.31.0.0/16 \ --token arnw7y.h3m0pggwza01avfc \ --apiserver-advertise-address 10.100.37.100 ``` ```shell= #結果 kubeadm join 10.100.37.50:6443 --token 7vgghl.rrgczyb24uqqgj0z --discovery-token-ca-cert-hash sha256:1aee2ef5cdf064f7280928eac18cb5c647eda293e6ff78566ba6d940572cae3d ``` ```shell= #找不到token: kubeadm token list #如果找不到hash值就: openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' ``` * 來看Node狀態 ``` $ mkdir ~/.kube && cp /etc/kubernetes/admin.conf ~/.kube/config $ kubectl get node NAME STATUS ROLES AGE VERSION master1 NotReady master 4m v1.9.6 ```  * init master success then dwonload CNI to create network for Node&Pod * Source file: * [Flannel](https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml)  * But flannel can't work we change CNI to [Calico](https://igene.tw/calico-architecture?fbclid=IwAR2Ezrg31zJTAk-f8Op18uDCwYcOKq2uD8QomYt87oUqjFcyk9vtkkBhL9U) * Source: * [Calico](https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml) //change CALICO_IPV4POOL_CIDR to 172.31.0.0/16 * [rbac](https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml)  * check routing rule with calico  * Check master node is Ready  * 退出sudo ```shell= $ sudo cp /etc/kubernetes/admin.conf $HOME/ $ sudo chown $(id -u):$(id -g) $HOME/admin.conf $ export KUBECONFIG=$HOME/admin.conf ``` * 讓Master也加入排成 ```shell= $ kubectl taint nodes --all node-role.kubernetes.io/master- ``` # Node加入 * 幫node安裝kubernetes ```shell= $ export KUBE_VERSION="1.13.4" $ sudo apt-get update && sudo apt-get install -y kubelet=${KUBE_VERSION}-00 kubeadm=${KUBE_VERSION}-00 kubernetes-cni=0.6.0-00 ``` * 完成後就可以開始加入 Node，這邊需要進入root使用者執行以下指令： ```shell= kubeadm join --token 4ixle5.p0oiqo1z9qtzwpp2 10.100.37.100:6443 --discovery-token-ca-cert-hash sha256:2041b4872d4b6ca80cf6bb71590076b515edf2796d6bbe7f56d6fd0a95bfe4b7 ``` * 回到master1查看節點狀態：  * Deploy web app on Node ```shell= $ kubectl create namespace sock-shop $ kubectl apply -n sock-shop -f "https://github.com/microservices-demo/microservices-demo/blob/master/deploy/kubernetes/complete-demo.yaml?raw=true" ``` * service Work  * Pod Create  * see the website  * 我們也可以起一個ubuntu ```shell= $ kubectl run my-shell -it --image ubuntu -- bash # 開完是沒有ifconfig還有ping 要另外抓 $ apt-get install iputils-ping -y $ apt-get install net-tools -y # 如果要刪除 就要把她的deployment刪除 $ kubectl get deployment --all-namespaces $ kubectl delete deployment XXXX ``` * 或是撰寫yaml ```yaml= apiVersion: v1 kind: Pod metadata: name: triple-pod1 spec: containers: - name: ubuntu image: ubuntu:v5 args: [bash, -c, 'for ((i = 0; ; i++)); do echo "$i: $(date)"; sleep 100; done'] ``` * 我們可能也需要安裝[Dashboard](https://hackmd.mcl.math.ncu.edu.tw/s/ryYuh8poV)  * Login方式 ```shell= $ kubectl -n kube-system get secret # All secrets with type 'kubernetes.io/service-account-token' will allow to log in. # Note that they have different privileges. NAME TYPE DATA AGE deployment-controller-token-frsqj kubernetes.io/service-account-token 3 22h $ kubectl -n kube-system describe secret deployment-controller-token-frsqj Name: deployment-controller-token-frsqj Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name=deployment-controller kubernetes.io/service-account.uid=64735958-ae9f-11e7-90d5-02420ac00002 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1025 bytes namespace: 11 bytes token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZXBsb3ltZW50LWNvbnRyb2xsZXItdG9rZW4tZnJzcWoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVwbG95bWVudC1jb250cm9sbGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNjQ3MzU5NTgtYWU5Zi0xMWU3LTkwZDUtMDI0MjBhYzAwMDAyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRlcGxveW1lbnQtY29udHJvbGxlciJ9.OqFc4CE1Kh6T3BTCR4XxDZR8gaF1MvH4M3ZHZeCGfO-sw-D0gp826vGPHr_0M66SkGaOmlsVHmP7zmTi-SJ3NCdVO5viHaVUwPJ62hx88_JPmSfD0KJJh6G5QokKfiO0WlGN7L1GgiZj18zgXVYaJShlBSz5qGRuGf0s1jy9KOBt9slAN5xQ9_b88amym2GIXoFyBsqymt5H-iMQaGP35tbRpewKKtly9LzIdrO23bDiZ1voc5QZeAZIWrizzjPY5HPM1qOqacaY9DcGc7akh98eBJG_4vZqH2gKy76fMf0yInFTeNKr45_6fWt8gRM77DQmPwb3hbrjWXe1VvXX_g ``` * 把節點刪除 ```shell= kubectl drain kube-node --delete-local-data --force --ignore-daemonsets kubectl delete node kube-node ``` * 由於k8s把一些敏感的system control value 給鎖起來 所以需要直接修改docker runtime ```shell= nsenter -t $(docker inspect --format '{{ .State.Pid }}' $(dockerID)) -n sysctl -w net.ipv4.ip_forward=1 or nsenter --target 1228034 --mount --uts --ipc --net --pid ```