LVS-fwm&persistence

Evernote Export

1、LVS-fwm

fwm：FireWall Mark

在netfilter上给报文打标记；mangle表；

ipvsadm -A|E -t|u|f service-address [-s scheduler]

-t, -u: service-address

ip:port

-f: service-address

firewall mark

iptables的功能：

filter, nat, mangle, raw

mangle：

target: MARK

–set-mark value[/mask]

基于fwm定义集群服务的步骤：

(1) 打标

# iptables -t mangle -A PREROUTING -d $vip -p $protocol –dport $serviceport -j MARK –set-mark #

(2) 定义集群服务

# ipvsadm -A -f # [-s scheduler]

实验拓扑：

DS地址：192.168.150.137 DIP、192.168.150.131 VIP

RS地址：192.168.150.138 RS1、192.168.150.139 RS1

CLIENT:192.168.150.133

LVS-fwm测试

DS配置（RS配置同LVS-dr配置）

[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp –dport 80 -j MARK –s et-mark 1 iptables上添加MARK规则，规则在PREROUTING上设置，设定在VIP上面

[root@localhost ~]# iptables -t mangle -vnL

Chain PREROUTING (policy ACCEPT 12 packets, 792 bytes)

pkts bytes target prot opt in out source destination

0 0 MARK tcp — * * 0.0.0.0/0 192.168.150.131 tcp dpt:80 M

ARK set 0x1

Chain INPUT (policy ACCEPT 12 packets, 792 bytes)

pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 7 packets, 1284 bytes)

pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 7 packets, 1284 bytes)

pkts bytes target prot opt in out source destination

[root@localhost ~]# iptables -t mangle -vnL

Chain PREROUTING (policy ACCEPT 20 packets, 1284 bytes)

pkts bytes target prot opt in out source destination

3 152 MARK tcp — * * 0.0.0.0/0 192.168.150.131 tcp dpt:80 M 当有客户端请求时，iptables开始打mark

ARK set 0x1

Chain INPUT (policy ACCEPT 20 packets, 1284 bytes)

pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 15 packets, 2656 bytes)

pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 15 packets, 2656 bytes)

pkts bytes target prot opt in out source destination

[root@localhost ~]# ipvsadm -A -f 1 -s rr 添加LVS集群服务 -f即为fwm -s调度方法为rr轮询

[root@localhost ~]# ipvsadm -a -f 1 -r 192.168.150.138 -g -w 1 管理集群服务上的RS -g为gateway默认模式

[root@localhost ~]# ipvsadm -a -f 1 -r 192.168.150.139 -g -w 1

[root@localhost ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

FWM 1 rr

-> 192.168.150.138:0 Route 1 0 0

-> 192.168.150.139:0 Route 1 0 0

[root@localhost ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

FWM 1 rr

-> 192.168.150.138:0 Route 1 0 0

-> 192.168.150.139:0 Route 1 0 0

此时的fwm已经配置完成并生效

客户端访问结果：

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS1</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS1</h1>

2、LVS persistent持久连接

功能：无论ipvs使用何种scheduler，其都能够实现在指定时间范围内始终将来自同一个ip地址的请求发往同一个RS；此功能是通过lvs持久连接模板实现，其与调度方法无关；

ipvs持久连接的模式：

每端口持久（PPC）:对某个端口进行持久

每客户端持久（PCC）：定义tcp或udp协议的0号端口为集群服务端口；

每FWM持久（PFWMC）：对防火墙编辑下的端口进行持久

http, https

定义持久连接服务的方法：

ipvsadm -A|E -t|u|f service-address [-s scheduler]

[-p [timeout]]

实验拓扑：

DS地址：192.168.150.137 DIP、192.168.150.131 VIP

RS地址：192.168.150.138 RS1、192.168.150.139 RS1

CLIENT:192.168.150.133

1、每客户端持久（PCC）：定义tcp或udp协议的0号端口为集群服务端口；

[root@localhost ~]# ipvsadm -E -f 1 -s rr -p 300 此时将此前的LVS集群服务加上-p进行持久

[root@localhost ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

FWM 1 rr persistent 300

-> 192.168.150.138:0 Route 1 0 0 添加时ip地址不加端口号指定默认为0，代表没客户端持久连接，客户端第一次连入此RS后，后面连接均在此RS上

-> 192.168.150.139:0 Route 1 0 0

客户端测试结果，调度算法无效果：

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

[root@localhost ~]# curl http://192.168.150.131

<h1>RS2</h1>

DS端也可以对ip进行:0指定进行设置

[root@localhost ~]# ipvsadm -A -t 192.168.150.131:0 -s rr -p

[root@localhost ~]# ipvsadm -a -t 192.168.150.131:0 -r 192.168.150.138 -g -w 1

[root@localhost ~]# ipvsadm -a -t 192.168.150.131:0 -r 192.168.150.139 -g -w 1

2、每端口持久（PPC）:对某个端口进行持久

[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp –dport 80 -j MARK –s et-mark 1

[root@localhost ~]# ipvsadm -A -t 192.168.150.131:23 -s rr -p 300 加入-p选项时即表明开启持久连接，此时持久连接为300s，地址后面加端口号表示仅对某个端口进行持久

[root@localhost ~]# ipvsadm -a -t 192.168.150.131:23 -r 192.168.150.138 -g -w 1

[root@localhost ~]# ipvsadm -a -t 192.168.150.131:23 -r 192.168.150.139 -g -w 1

RS端开启telnet服务1和2均开启

[root@localhost ~]# systemctl start telnet.socket

[root@localhost ~]# ss -tnl

State Recv-Q Send-Q Local Address:Port Peer Address:Port

LISTEN 0 128 *:22 *:*

LISTEN 0 100 127.0.0.1:25 *:*

LISTEN 0 128 :::80 :::*

LISTEN 0 128 :::22 :::*

LISTEN 0 128 :::23 :::*

LISTEN 0 100 ::1:25 :::*

客户端测试结果：

[root@localhost ~]# ssh 192.168.150.131 ssh访问会持久在192.168.150.139这台RS上

The authenticity of host '192.168.150.131 (192.168.150.131)' can't be established.

RSA key fingerprint is 22:fc:db:5b:e5:26:8a:35:96:9f:2d:c4:4f:07:d1:e8.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '192.168.150.131' (RSA) to the list of known hosts.

root@192.168.150.131's password:

Permission denied, please try again.

root@192.168.150.131's password:

Last failed login: Thu Dec 1 01:23:09 CST 2016 from 192.168.150.133 on ssh:notty

There were 2 failed login attempts since the last successful login.

Last login: Thu Dec 1 00:34:06 2016 from 192.168.150.1

[root@localhost ~]# ifconfig

eno33554976: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.150.139 netmask 255.255.255.0 broadcast 192.168.150.255

inet6 fe80::20c:29ff:fe7c:2ca9 prefixlen 64 scopeid 0x20<link>

ether 00:0c:29:7c:2c:a9 txqueuelen 1000 (Ethernet)

RX packets 8532 bytes 8978251 (8.5 MiB)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 3594 bytes 320788 (313.2 KiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

inet 127.0.0.1 netmask 255.0.0.0

inet6 ::1 prefixlen 128 scopeid 0x10<host>

loop txqueuelen 0 (Local Loopback)

RX packets 20 bytes 1760 (1.7 KiB)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 20 bytes 1760 (1.7 KiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

inet 192.168.150.131 netmask 255.255.255.255

loop txqueuelen 0 (Local Loopback)

3、对于多端口的持久可以通过使用FWM中的防火墙下的端口进行持久

测试：进行http和https的端口持久

DS主机配置：

情况规则

root@localhost ~]# ipvsadm -C

创建私有CA

[root@localhost ~]# cd /etc/pki/CA/

[root@localhost CA]# (umask 077;openssl genrsa -out private/cakey.pem 2048)

Generating RSA private key, 2048 bit long modulus

…………………………………………+++

……..+++

e is 65537 (0x10001)

[root@localhost CA]# ls -l private/

总用量 4

-rw——- 1 root root 1679 12月 1 01:28 cakey.pem

[root@localhost CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem -days 365

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

—–

Country Name (2 letter code) [XX]:CN

State or Province Name (full name) []:SZ

Locality Name (eg, city) [Default City]:SZ

Organization Name (eg, company) [Default Company Ltd]:CPTW

Organizational Unit Name (eg, section) []:OPS

Common Name (eg, your name or your server's hostname) []:CPTW.COM.CN

Email Address []:MAIL.CPTW.COM.CN

[root@localhost CA]# touch index.txt

[root@localhost CA]# echo 01 > serial

RS1配置

包安装，RS2上同步安装

[root@localhost ~]# yum -y install mod_ssl

已加载插件：fastestmirror

Loading mirror speeds from cached hostfile

* base: mirrors.cn99.com

* extras: mirrors.cn99.com

* updates: mirrors.cn99.com

正在解决依赖关系

–> 正在检查事务

—> 软件包 mod_ssl.x86_64.1.2.4.6-40.el7.centos.4 将被安装

–> 解决依赖关系完成

依赖关系解决

=====================================================================================================

Package 架构版本源大小

=====================================================================================================

正在安装:

mod_ssl x86_64 1:2.4.6-40.el7.centos.4 updates 104 k

事务概要

=====================================================================================================

安装 1 软件包

总下载量：104 k

安装大小：224 k

Downloading packages:

mod_ssl-2.4.6-40.el7.centos.4.x86_64.rpm | 104 kB 00:00:00

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

正在安装 : 1:mod_ssl-2.4.6-40.el7.centos.4.x86_64 1/1

验证中 : 1:mod_ssl-2.4.6-40.el7.centos.4.x86_64 1/1

已安装:

mod_ssl.x86_64 1:2.4.6-40.el7.centos.4

完毕！

秘钥加密

[root@localhost ~]# cd /etc/httpd/

[root@localhost httpd]# ls

conf conf.d conf.modules.d logs modules run

[root@localhost httpd]# mkdir ssl

[root@localhost httpd]# cd ssl

[root@localhost ssl]# ls

[root@localhost ssl]# (umask 077;openssl genrsa -out httpd.key 1024)

Generating RSA private key, 1024 bit long modulus

……..++++++

……..++++++

e is 65537 (0x10001)

[root@localhost ssl]# ls

httpd.key

证书请求

[root@localhost ssl]# openssl req -new -key httpd.key -out httpd.csr

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

—–

Country Name (2 letter code) [XX]:CN

State or Province Name (full name) []:SZ

Locality Name (eg, city) [Default City]:SZ

Organization Name (eg, company) [Default Company Ltd]:CPTW

Organizational Unit Name (eg, section) []:OPS

Common Name (eg, your name or your server's hostname) []:WWW.CPTW.COM.CN

Email Address []:MAIL.CPTW.COM.CN

Please enter the following 'extra' attributes

to be sent with your certificate request

A challenge password []:

An optional company name []:

[root@localhost ssl]# ls

httpd.csr httpd.key root@192.168.150.137

将证书请求发送至颁发证书机构进行签证

[root@localhost ssl]# scp httpd.csr root@192.168.150.137:/tmp

root@192.168.150.137's password:

httpd.csr

DS上进行证书签证并回拷至RS1

[root@localhost tmp]# ls

httpd.crt httpd.csr ipvsadm-config.v1

[root@localhost tmp]# openssl ca -in /tmp/httpd.csr -out /tmp/httpd.crt -days 365

Using configuration from /etc/pki/tls/openssl.cnf

Check that the request matches the signature

Signature ok

Certificate Details:

Serial Number: 1 (0x1)

Validity

Not Before: Nov 30 18:08:27 2016 GMT

Not After : Nov 30 18:08:27 2017 GMT

Subject:

countryName = CN

stateOrProvinceName = SZ

organizationName = CPTW

organizationalUnitName = OPS

commonName = WWW.CPTW.COM.CN

emailAddress = MAIL.CPTW.COM.CN

X509v3 extensions:

X509v3 Basic Constraints:

CA:FALSE

Netscape Comment:

OpenSSL Generated Certificate

X509v3 Subject Key Identifier:

C2:C2:F3:E3:91:CC:82:96:B2:38:CB:23:84:F4:4F:93:FF:32:FC:BD

X509v3 Authority Key Identifier:

keyid:F0:26:D0:A8:94:A9:81:E2:C5:C0:5A:95:5B:D3:1B:BB:BB:28:59:87

Certificate is to be certified until Nov 30 18:08:27 2017 GMT (365 days)

Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y

Write out database with 1 new entries

Data Base Updated

[root@localhost tmp]# ls

httpd.crt httpd.csr ipvsadm-config.v1

[root@localhost tmp]# scp httpd.crt root@192.168.150.138:/etc/httpd/ssl

root@192.168.150.138's password:

httpd.crt

RS1修改httpd相关配置，并将证书和配置拷贝至RS2

[root@localhost httpd]# vim conf.d/ssl.conf

SSLCertificateFile /etc/httpd/ssl/httpd.crt

SSLCertificateKeyFile /etc/httpd/ssl/httpd.key

[root@localhost ssl]# scp -rp /etc/httpd/ssl/ root@192.168.150.139:/etc/httpd/

root@192.168.150.139's password:

httpd.key 100% 887 0.9KB/s 00:00

root@192.168.150.137 100% 676 0.7KB/s 00:00

httpd.csr 100% 684 0.7KB/s 00:00

httpd.crt 100% 3808 3.7KB/s 00:00

[root@localhost ssl]# scp /etc/httpd/conf.d/ssl.conf root@192.168.150.139:/etc/httpd/conf.d/ssl.conf

root@192.168.150.139's password:

ssl.conf

两台RS均进行httpd的重启并查看443是否已正常开启

[root@localhost httpd]# systemctl start httpd.service

[root@localhost httpd]# ss -tnl

State Recv-Q Send-Q Local Address:Port Peer Address:Port

LISTEN 0 128 *:22 *:*

LISTEN 0 100 127.0.0.1:25 *:*

LISTEN 0 128 :::80 :::*

LISTEN 0 128 :::22 :::*

LISTEN 0 128 :::23 :::*

LISTEN 0 100 ::1:25 :::*

LISTEN 0 128 :::443 :::*

DS主机配置

[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp -m multiport –dports 80,443 -j MARK –set-mark 10 添加了80和443两个目标端口并mark

[root@localhost ~]# iptables -t mangle -vnL

Chain PREROUTING (policy ACCEPT 135 packets, 11432 bytes)

pkts bytes target prot opt in out source destination

0 0 MARK tcp — * * 0.0.0.0/0 192.168.150.131 multiport dp

orts 80,443 MARK set 0xa

Chain INPUT (policy ACCEPT 135 packets, 11432 bytes)

pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 117 packets, 11516 bytes)

pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 117 packets, 11516 bytes)

pkts bytes target prot opt in out source destination

客户端进行请求

[root@localhost ~]# curl http://192.168.150.131

curl: (7) couldn't connect to host

[root@localhost ~]# curl https://192.168.150.131

curl: (7) couldn't connect to host

[root@localhost ~]# curl https://192.168.150.138

curl: (60) Peer certificate cannot be authenticated with known CA certificates

此时DS上状态，已经有包请求并以mark

[root@localhost ~]# iptables -t mangle -vnL

Chain PREROUTING (policy ACCEPT 142 packets, 11892 bytes)

pkts bytes target prot opt in out source destination

2 120 MARK tcp — * * 0.0.0.0/0 192.168.150.131 multiport dp

orts 80,443 MARK set 0xa

Chain INPUT (policy ACCEPT 142 packets, 11892 bytes)

pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 124 packets, 12864 bytes)

pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 124 packets, 12864 bytes)

pkts bytes target prot opt in out source destination

DS上添加持久

[root@localhost ~]# ipvsadm -A -f 10 -s rr -p

[root@localhost ~]# ipvsadm -a -f 10 -r 192.168.150.138 -g -w 1

[root@localhost ~]# ipvsadm -a -f 10 -r 192.168.150.139 -g -w 2

[root@localhost ~]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

FWM 10 rr persistent 360

-> 192.168.150.138:0 Route 1 0 0

-> 192.168.150.139:0 Route 2 0 0

客户端执行结果：此时客户端访问的443和80端口均会绑定持久在同一台RS上

[root@localhost ~]# curl http://192.168.150.131

<h1>RS1</h1>

[root@localhost ~]# curl -k https://192.168.150.131

<h1>RS1</h1>