MENU

centos 安装 flannel

• 2019 年 06 月 11 日 • linux,kubernetes

预估

flannel架构介绍

flannel如何将一个pod里面的流量转发到另一台物理机里面的pod?

1.容器直接使用目标容器的ip访问,默认通过容器内部的eth0发送出去。
2.报文通过veth pair被发送到vethXXX。
3.vethXXX是直接连接到虚拟交换机docker0的,报文通过虚拟bridge docker0发送出去。
4.查找路由表,外部容器ip的报文都会转发到flannel0虚拟网卡,这是一个P2P的虚拟网卡,然后报文就被转发到监听在另一端的flanneld。
5.flanneld通过etcd维护了各个节点之间的路由表,把原来的报文UDP封装一层,通过配置的iface发送出去。
6.报文通过主机之间的网络找到目标主机。
7.报文继续往上,到传输层,交给监听在8285端口的flanneld程序处理。
8.数据被解包,然后发送给flannel0虚拟网卡。
9.查找路由表,发现对应容器的报文要交给docker0。
10.docker0找到连到自己的容器,把报文发送过去。

flannel安装配置

安装etcd

flannel使用的配置信息,比如网络的地址,掩码存储才etcd里面,所以需要先安装etcd。

安装flanned

官网地址是: https://github.com/coreos/flannel
最新的下载地址是 https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz

二进制方式下载并安装flannel

[jjzhong@k8s-node1 flannel]$ wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
[jjzhong@k8s-node1 flannel]$ tar -xvf flannel-v0.11.0-linux-amd64.tar.gz 
flanneld
mk-docker-opts.sh
README.md
[jjzhong@k8s-node1 flannel]$ sudo cp flanneld /usr/local/bin/

//在etcd中配置好网络信息

[jjzhong@k8s-node1 flannel]$ etcdctl --endpoints "http://172.30.252.157:2379" \
set /coreos.com/network/config '{"NetWork":"10.0.0.0/16", "Backend": {"Type": "udp"}}'

如果使用wxlan方式则使用
etcdctl --endpoints "http://172.30.252.157:2379" \
set /coreos.com/network/config '{"NetWork":"10.88.0.0/16", "Backend": {"Type": "wxlan"}}'

增加一个systemd启动文件,iface 修改为当前机器的物理网卡ip

$ cat <<EOF | sudo tee /etc/systemd/system/flanneld.service
[Unit]
Description=Flanneld
Documentation=https://github.com/coreos/flannel
After=network.target
Before=docker.service

[Service]
User=root
ExecStart=/usr/local/bin/flanneld \
--etcd-endpoints="http://172.30.252.157:2379" \
--iface=172.30.252.157 \
--ip-masq
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

启动flanneld,并设置为开机启动方式

sudo systemctl daemon-reload
sudo systemctl start flanneld
sudo systemctl enable flanneld

查看flannel运行状态

[jjzhong@k8s-node1 flannel]$ sudo systemctl status flanneld
● flanneld.service - Flanneld
   Loaded: loaded (/etc/systemd/system/flanneld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-06-11 11:40:36 CST; 18s ago
     Docs: https://github.com/coreos/flannel
 Main PID: 20012 (flanneld)
   CGroup: /system.slice/flanneld.service
           └─20012 /usr/local/bin/flanneld --etcd-endpoints=http://172.30.252.157:2379 --iface=172.30.251.33 --ip-masq

Jun 11 11:40:36 k8s-node1 flanneld[20012]: I0611 11:40:36.158673   20012 iptables.go:167] Deleting iptables rule: -s 10.88.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
Jun 11 11:40:36 k8s-node1 flanneld[20012]: I0611 11:40:36.160949   20012 iptables.go:167] Deleting iptables rule: -d 10.88.0.0/16 -j ACCEPT
Jun 11 11:40:36 k8s-node1 flanneld[20012]: I0611 11:40:36.162880   20012 iptables.go:167] Deleting iptables rule: ! -s 10.88.0.0/16 -d 10.88.81.0/24 -j RETURN
Jun 11 11:40:36 k8s-node1 flanneld[20012]: I0611 11:40:36.165219   20012 iptables.go:155] Adding iptables rule: -s 10.88.0.0/16 -j ACCEPT
Jun 11 11:40:36 k8s-node1 flanneld[20012]: I0611 11:40:36.168235   20012 iptables.go:167] Deleting iptables rule: ! -s 10.88.0.0/16 -d 10.88.0.0/16 -j MASQUERADE
Jun 11 11:40:36 k8s-node1 flanneld[20012]: I0611 11:40:36.173115   20012 iptables.go:155] Adding iptables rule: -s 10.88.0.0/16 -d 10.88.0.0/16 -j RETURN
Jun 11 11:40:36 k8s-node1 flanneld[20012]: I0611 11:40:36.174603   20012 iptables.go:155] Adding iptables rule: -d 10.88.0.0/16 -j ACCEPT
Jun 11 11:40:36 k8s-node1 flanneld[20012]: I0611 11:40:36.198196   20012 iptables.go:155] Adding iptables rule: -s 10.88.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
Jun 11 11:40:36 k8s-node1 flanneld[20012]: I0611 11:40:36.207051   20012 iptables.go:155] Adding iptables rule: ! -s 10.88.0.0/16 -d 10.88.81.0/24 -j RETURN
Jun 11 11:40:36 k8s-node1 flanneld[20012]: I0611 11:40:36.215909   20012 iptables.go:155] Adding iptables rule: ! -s 10.88.0.0/16 -d 10.88.0.0/16 -j MASQUERADE

查看etcd中存储的网络信息

[netpay@k8s-master ~]$ etcdctl --endpoints "http://172.30.252.157:2379" ls /coreos.com/network/subnets 
/coreos.com/network/subnets/10.88.81.0-24

查看机器的网络信息,flannel0

[jjzhong@k8s-node1 flannel]$ ifconfig flannel0
flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1472
        inet 10.88.81.0  netmask 255.255.255.255  destination 10.88.81.0
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0    

flannel启动过程解析

flannel服务需要先于Docker启动。flannel服务启动时主要做了以下几步的工作:

1.从etcd中获取network的配置信息。
2.划分subnet,并在etcd中进行注册。
3.将子网信息记录到/run/flannel/subnet.env中。

配置docker

flannel运行后会自动生成一个文件 /run/flannel/subnet.env

[jjzhong@k8s-node1 flannel]$ cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.88.0.0/16
FLANNEL_SUBNET=10.88.81.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=true

产生这个文件唯一的作用就是告诉docker来读取这个文件,并按照这个文件的内容来设置docker的启动参数。

创建docker运行参数

flannel生产了相关的网络配置参数,但是和docker所需要的参数格式还是有区别的,所以需要使用flannel提供的脚本来转换一下

[jjzhong@k8s-node1 flannel]$ sudo ./mk-docker-opts.sh -d /run/docker_opts.env -c
[jjzhong@k8s-node1 flannel]$ cat /run/docker_opts.env
DOCKER_OPTS=" --bip=10.88.81.1/24 --ip-masq=false --mtu=1472"

上面配置需要手动执行,为了flanneld启动时就自动设置,将相关命令增加到flanneld的启动脚本中

[jjzhong@k8s-node1 flannel]$ sudo mv mk-docker-opts.sh /usr/local/bin


[Unit]
Description=Flanneld
Documentation=https://github.com/coreos/flannel
After=network.target
Before=docker.service

[Service]
User=root
ExecStart=/usr/local/bin/flanneld --etcd-endpoints=http://172.30.252.157:2379 --iface=172.30.251.33 --ip-masq
#新增下面这个配置
ExecStartPost=/usr/local/bin/mk-docker-opts.sh  -d /run/docker_opts.env
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

修改docker的启动参数,让它能够读取这些网络配置

[jjzhong@k8s-node1 flannel]$ vi /usr/lib/systemd/system/docker.service 

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
#在docker后启动,以便于将flanneld设置的网络信息给docker使用
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
# 下面一行 和 $DOCKER_OPTS 是增加的
EnvironmentFile=-/run/docker_opts.env
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock $DOCKER_OPTS
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

StartLimitBurst=3
StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this option.
TasksMax=infinity

# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes

# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

可以看到docker0 和flanneld在同一个网络地址空间了

[jjzhong@k8s-node1 flannel]$ ifconfig flannel0
flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1472
        inet 10.88.81.0  netmask 255.255.255.255  destination 10.88.81.0
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[jjzhong@k8s-node1 flannel]$ ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 10.88.81.1  netmask 255.255.255.0  broadcast 10.88.81.255
        inet6 fe80::42:efff:fe60:c683  prefixlen 64  scopeid 0x20<link>
        ether 02:42:ef:60:c6:83  txqueuelen 0  (Ethernet)
        RX packets 691577  bytes 186074316 (177.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 771872  bytes 86074619 (82.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

请输入图片描述
请输入图片描述

最后编辑于: 2019 年 08 月 04 日