Hexo

CentOS安装LVS及长连接配置

先安装如下软件

yum install ipvsadm

yum install keepalived

修改配置/etc/sysctl.conf中，将net.ipv4.ip_forward配置为1：

net.ipv4.ip_forward = 1

使用sysctl -p让配置生效

##NAT模式

NAT模式

配置/etc/keepalived/keepalived.conf文件：

! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_MASTER
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.222
    }
}


virtual_server 192.168.1.222 20000{
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    persistence_timeout 7200
    protocol TCP

    real_server 192.168.1.203 20000 {
        weight 3
        TCP_CHECK {  
            connect_timeout 3  
            nb_get_retry 3  
            delay_before_retry 3
            connect_port 20000
        }  
    }

   real_server 192.168.1.204 20000 {
        weight 3
        TCP_CHECK {  
            connect_timeout 3  
            nb_get_retry 3  
            delay_before_retry 3
            connect_port 20000
        }  
    }

}

将2台real server(192.168.1.203, 192.168.1.204)网关配置为192.168.1.222

启动keepalived

1
service keepalived start

通过ipvsadm查看连接状况

$ ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.1.210:dnp rr persistent 7200
  -> 192.168.1.203:dnp            Masq    3      0          0  
  -> 192.168.1.204:dnp            Masq    3      0          0

LVS & keepalived的tcp长连接Connection reset by peer错误

查看tcp session的超时时间，如果设置比较短，则会报错
ipvsadm –list –timeout
Timeout (tcp tcpfin udp): 900 120 300
表示tcp session的timeout是900秒

通过–set可以设置timeout时间
ipvsadm –set 7200 120 300

keepalived配置中virtual_server的persistence_timeout, 对于长连接应该配置长一些，可以和LVS的tcp timeout配置一直

virtual_server 192.168.1.210 20000{
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    persistence_timeout 7200
    protocol TCP

##Direct Routing模式

理解DR模式的原理非常重要，这样就知道为什么需要封堵ARP消息了

DR原理

可以看出，在返回消息中，返回路径和请求路径不等同，所以需要在RealServer的lo接口上，加上VIP地址

! Configuration File for keepalived

global_defs {
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_MASTER
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    #advert_int 1
    #authentication {
    #    auth_type PASS
    #    auth_pass 1111
    #}
    virtual_ipaddress {
        192.168.1.222
    }
}

virtual_server 192.168.1.222 80{
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    nat_mask 255.255.255.0
    persistence_timeout 7200
    protocol TCP

   real_server 192.168.1.203 80{
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
   }

   real_server 192.168.1.204 80{
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        connect_port 80
        }  
    }

}

同时在RealServer 192.168.1.203, 192.168.1.204上执行如下指令：

#!/bin/sh

SNS_VIP=192.168.1.222
. /etc/rc.d/init.d/functions  

case "$1" in  
start)  
  ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP  
  /sbin/route add -host $SNS_VIP dev lo:0  
  echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore  
  echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce  
  echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore  
  echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce  
  sysctl -p >/dev/null 2>&1  
  echo "RealServer Start OK"  
;;  
stop)  
  ifconfig lo:0 down  
  route del $SNS_VIP >/dev/null 2>&1  
  echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore  
  echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce  
  echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore  
  echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce  
  echo "RealServer Stoped"  
;;  
*)  
  echo "Usage: $0 {start|stop}"  
  exit 1  
esac  
exit 0

通过ipvsadm -lcn可以查看连接情况，如果出现SYN_RECV状态，多半是ARP问题，请检查ARP或网关是否正确.

可以通过命令直接增加lvs，类似如下

ifconfig eth0:0 192.168.1.100 netmask 255.255.255.0 broadcast 192.168.1.255 up
ipvsadm -C  #清除
ipvsadm -A -t 192.168.1.100:80 -s wlc
ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.206:80 -g
ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.207:3636 -g
lvs centoslvs