1、LVS-fwm
fwm：FireWall Mark
在netfilter上给报文打标记；mangle表；
ipvsadm -A|E -t|u|f service-address [-s scheduler]
-t, -u: service-address
ip:port
-f: service-address
firewall mark
iptables的功能：
filter, nat, mangle, raw
mangle：
target: MARK
–set-mark value[/mask]
基于fwm定义集群服务的步骤：
(1) 打标
# iptables -t mangle -A PREROUTING -d $vip -p $protocol –dport $serviceport -j MARK –set-mark #
(2) 定义集群服务
# ipvsadm -A -f # [-s scheduler]
实验拓扑：
DS地址：192.168.150.137 DIP、192.168.150.131 VIP
RS地址：192.168.150.138 RS1、192.168.150.139 RS1
CLIENT:192.168.150.133
LVS-fwm测试
DS配置（RS配置同LVS-dr配置）
[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp –dport 80 -j MARK –s et-mark 1 iptables上添加MARK规则，规则在PREROUTING上设置，设定在VIP上面
[root@localhost ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 12 packets, 792 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK tcp — * * 0.0.0.0/0 192.168.150.131 tcp dpt:80 M
ARK set 0x1
Chain INPUT (policy ACCEPT 12 packets, 792 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 7 packets, 1284 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 7 packets, 1284 bytes)
pkts bytes target prot opt in out source destination
[root@localhost ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 20 packets, 1284 bytes)
pkts bytes target prot opt in out source destination
3 152 MARK tcp — * * 0.0.0.0/0 192.168.150.131 tcp dpt:80 M 当有客户端请求时，iptables开始打mark
ARK set 0x1
Chain INPUT (policy ACCEPT 20 packets, 1284 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 15 packets, 2656 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 15 packets, 2656 bytes)
pkts bytes target prot opt in out source destination
[root@localhost ~]# ipvsadm -A -f 1 -s rr 添加LVS集群服务 -f即为fwm -s调度方法为rr轮询
[root@localhost ~]# ipvsadm -a -f 1 -r 192.168.150.138 -g -w 1 管理集群服务上的RS -g为gateway默认模式
[root@localhost ~]# ipvsadm -a -f 1 -r 192.168.150.139 -g -w 1
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 rr
-> 192.168.150.138:0 Route 1 0 0
-> 192.168.150.139:0 Route 1 0 0
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 rr
-> 192.168.150.138:0 Route 1 0 0
-> 192.168.150.139:0 Route 1 0 0
此时的fwm已经配置完成并生效
客户端访问结果：
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS1</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS1</h1>
2、LVS persistent持久连接
功能：无论ipvs使用何种scheduler，其都能够实现在指定时间范围内始终将来自同一个ip地址的请求发往同一个RS；此功能是通过lvs持久连接模板实现，其与调度方法无关；
ipvs持久连接的模式：
每端口持久（PPC）:对某个端口进行持久
每客户端持久（PCC）：定义tcp或udp协议的0号端口为集群服务端口；
每FWM持久（PFWMC）：对防火墙编辑下的端口进行持久
http, https
定义持久连接服务的方法：
ipvsadm -A|E -t|u|f service-address [-s scheduler]
[-p [timeout]]
实验拓扑：
DS地址：192.168.150.137 DIP、192.168.150.131 VIP
RS地址：192.168.150.138 RS1、192.168.150.139 RS1
CLIENT:192.168.150.133
1、每客户端持久（PCC）：定义tcp或udp协议的0号端口为集群服务端口；
[root@localhost ~]# ipvsadm -E -f 1 -s rr -p 300 此时将此前的LVS集群服务加上-p进行持久
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 rr persistent 300
-> 192.168.150.138:0 Route 1 0 0 添加时ip地址不加端口号指定默认为0，代表没客户端持久连接，客户端第一次连入此RS后，后面连接均在此RS上
-> 192.168.150.139:0 Route 1 0 0
客户端测试结果，调度算法无效果：
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
[root@localhost ~]# curl http://192.168.150.131
<h1>RS2</h1>
DS端也可以对ip进行:0指定进行设置
[root@localhost ~]# ipvsadm -A -t 192.168.150.131:0 -s rr -p
[root@localhost ~]# ipvsadm -a -t 192.168.150.131:0 -r 192.168.150.138 -g -w 1
[root@localhost ~]# ipvsadm -a -t 192.168.150.131:0 -r 192.168.150.139 -g -w 1
2、每端口持久（PPC）:对某个端口进行持久
[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp –dport 80 -j MARK –s et-mark 1
[root@localhost ~]# ipvsadm -A -t 192.168.150.131:23 -s rr -p 300 加入-p选项时即表明开启持久连接，此时持久连接为300s，地址后面加端口号表示仅对某个端口进行持久
[root@localhost ~]# ipvsadm -a -t 192.168.150.131:23 -r 192.168.150.138 -g -w 1
[root@localhost ~]# ipvsadm -a -t 192.168.150.131:23 -r 192.168.150.139 -g -w 1
RS端开启telnet服务1和2均开启
[root@localhost ~]# systemctl start telnet.socket
[root@localhost ~]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::80 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 :::23 :::*
LISTEN 0 100 ::1:25 :::*
客户端测试结果：
[root@localhost ~]# ssh 192.168.150.131 ssh访问会持久在192.168.150.139这台RS上
The authenticity of host '192.168.150.131 (192.168.150.131)' can't be established.
RSA key fingerprint is 22:fc:db:5b:e5:26:8a:35:96:9f:2d:c4:4f:07:d1:e8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.150.131' (RSA) to the list of known hosts.
root@192.168.150.131's password:
Permission denied, please try again.
root@192.168.150.131's password:
Last failed login: Thu Dec 1 01:23:09 CST 2016 from 192.168.150.133 on ssh:notty
There were 2 failed login attempts since the last successful login.
Last login: Thu Dec 1 00:34:06 2016 from 192.168.150.1
[root@localhost ~]# ifconfig
eno33554976: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.150.139 netmask 255.255.255.0 broadcast 192.168.150.255
inet6 fe80::20c:29ff:fe7c:2ca9 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:7c:2c:a9 txqueuelen 1000 (Ethernet)
RX packets 8532 bytes 8978251 (8.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3594 bytes 320788 (313.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 20 bytes 1760 (1.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20 bytes 1760 (1.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.150.131 netmask 255.255.255.255
loop txqueuelen 0 (Local Loopback)
3、对于多端口的持久可以通过使用FWM中的防火墙下的端口进行持久
测试：进行http和https的端口持久
DS主机配置：
情况规则
root@localhost ~]# ipvsadm -C
创建私有CA
[root@localhost ~]# cd /etc/pki/CA/
[root@localhost CA]# (umask 077;openssl genrsa -out private/cakey.pem 2048)
Generating RSA private key, 2048 bit long modulus
…………………………………………+++
……..+++
e is 65537 (0x10001)
[root@localhost CA]# ls -l private/
总用量 4
-rw——- 1 root root 1679 12月 1 01:28 cakey.pem
[root@localhost CA]# openssl req -new -x509 -key private/cakey.pem -out cacert.pem -days 365
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
—–
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:SZ
Locality Name (eg, city) [Default City]:SZ
Organization Name (eg, company) [Default Company Ltd]:CPTW
Organizational Unit Name (eg, section) []:OPS
Common Name (eg, your name or your server's hostname) []:CPTW.COM.CN
Email Address []:MAIL.CPTW.COM.CN
[root@localhost CA]# touch index.txt
[root@localhost CA]# echo 01 > serial
RS1配置
包安装，RS2上同步安装
[root@localhost ~]# yum -y install mod_ssl
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* extras: mirrors.cn99.com
* updates: mirrors.cn99.com
正在解决依赖关系
–> 正在检查事务
—> 软件包 mod_ssl.x86_64.1.2.4.6-40.el7.centos.4 将被 安装
–> 解决依赖关系完成
依赖关系解决
=====================================================================================================
Package 架构 版本 源 大小
=====================================================================================================
正在安装:
mod_ssl x86_64 1:2.4.6-40.el7.centos.4 updates 104 k
事务概要
=====================================================================================================
安装 1 软件包
总下载量：104 k
安装大小：224 k
Downloading packages:
mod_ssl-2.4.6-40.el7.centos.4.x86_64.rpm | 104 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
正在安装 : 1:mod_ssl-2.4.6-40.el7.centos.4.x86_64 1/1
验证中 : 1:mod_ssl-2.4.6-40.el7.centos.4.x86_64 1/1
已安装:
mod_ssl.x86_64 1:2.4.6-40.el7.centos.4
完毕！
秘钥加密
[root@localhost ~]# cd /etc/httpd/
[root@localhost httpd]# ls
conf conf.d conf.modules.d logs modules run
[root@localhost httpd]# mkdir ssl
[root@localhost httpd]# cd ssl
[root@localhost ssl]# ls
[root@localhost ssl]# (umask 077;openssl genrsa -out httpd.key 1024)
Generating RSA private key, 1024 bit long modulus
……..++++++
……..++++++
e is 65537 (0x10001)
[root@localhost ssl]# ls
httpd.key
证书请求
[root@localhost ssl]# openssl req -new -key httpd.key -out httpd.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
—–
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:SZ
Locality Name (eg, city) [Default City]:SZ
Organization Name (eg, company) [Default Company Ltd]:CPTW
Organizational Unit Name (eg, section) []:OPS
Common Name (eg, your name or your server's hostname) []:WWW.CPTW.COM.CN
Email Address []:MAIL.CPTW.COM.CN
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@localhost ssl]# ls
httpd.csr httpd.key root@192.168.150.137
将证书请求发送至颁发证书机构进行签证
[root@localhost ssl]# scp httpd.csr root@192.168.150.137:/tmp
root@192.168.150.137's password:
httpd.csr
DS上进行证书签证并回拷至RS1
[root@localhost tmp]# ls
httpd.crt httpd.csr ipvsadm-config.v1
[root@localhost tmp]# openssl ca -in /tmp/httpd.csr -out /tmp/httpd.crt -days 365
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: Nov 30 18:08:27 2016 GMT
Not After : Nov 30 18:08:27 2017 GMT
Subject:
countryName = CN
stateOrProvinceName = SZ
organizationName = CPTW
organizationalUnitName = OPS
commonName = WWW.CPTW.COM.CN
emailAddress = MAIL.CPTW.COM.CN
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
C2:C2:F3:E3:91:CC:82:96:B2:38:CB:23:84:F4:4F:93:FF:32:FC:BD
X509v3 Authority Key Identifier:
keyid:F0:26:D0:A8:94:A9:81:E2:C5:C0:5A:95:5B:D3:1B:BB:BB:28:59:87
Certificate is to be certified until Nov 30 18:08:27 2017 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
[root@localhost tmp]# ls
httpd.crt httpd.csr ipvsadm-config.v1
[root@localhost tmp]# scp httpd.crt root@192.168.150.138:/etc/httpd/ssl
root@192.168.150.138's password:
httpd.crt
RS1修改httpd相关配置，并将证书和配置拷贝至RS2
[root@localhost httpd]# vim conf.d/ssl.conf
SSLCertificateFile /etc/httpd/ssl/httpd.crt
SSLCertificateKeyFile /etc/httpd/ssl/httpd.key
[root@localhost ssl]# scp -rp /etc/httpd/ssl/ root@192.168.150.139:/etc/httpd/
root@192.168.150.139's password:
httpd.key 100% 887 0.9KB/s 00:00
root@192.168.150.137 100% 676 0.7KB/s 00:00
httpd.csr 100% 684 0.7KB/s 00:00
httpd.crt 100% 3808 3.7KB/s 00:00
[root@localhost ssl]# scp /etc/httpd/conf.d/ssl.conf root@192.168.150.139:/etc/httpd/conf.d/ssl.conf
root@192.168.150.139's password:
ssl.conf
两台RS均进行httpd的重启并查看443是否已正常开启
[root@localhost httpd]# systemctl start httpd.service
[root@localhost httpd]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 :::80 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 :::23 :::*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 128 :::443 :::*
DS主机配置
[root@localhost ~]# iptables -t mangle -A PREROUTING -d 192.168.150.131 -p tcp -m multiport –dports 80,443 -j MARK –set-mark 10 添加了80和443两个目标端口并mark
[root@localhost ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 135 packets, 11432 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK tcp — * * 0.0.0.0/0 192.168.150.131 multiport dp
orts 80,443 MARK set 0xa
Chain INPUT (policy ACCEPT 135 packets, 11432 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 117 packets, 11516 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 117 packets, 11516 bytes)
pkts bytes target prot opt in out source destination
客户端进行请求
[root@localhost ~]# curl http://192.168.150.131
curl: (7) couldn't connect to host
[root@localhost ~]# curl https://192.168.150.131
curl: (7) couldn't connect to host
[root@localhost ~]# curl https://192.168.150.138
curl: (60) Peer certificate cannot be authenticated with known CA certificates
此时DS上状态，已经有包请求并以mark
[root@localhost ~]# iptables -t mangle -vnL
Chain PREROUTING (policy ACCEPT 142 packets, 11892 bytes)
pkts bytes target prot opt in out source destination
2 120 MARK tcp — * * 0.0.0.0/0 192.168.150.131 multiport dp
orts 80,443 MARK set 0xa
Chain INPUT (policy ACCEPT 142 packets, 11892 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 124 packets, 12864 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 124 packets, 12864 bytes)
pkts bytes target prot opt in out source destination
DS上添加持久
[root@localhost ~]# ipvsadm -A -f 10 -s rr -p
[root@localhost ~]# ipvsadm -a -f 10 -r 192.168.150.138 -g -w 1
[root@localhost ~]# ipvsadm -a -f 10 -r 192.168.150.139 -g -w 2
[root@localhost ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 10 rr persistent 360
-> 192.168.150.138:0 Route 1 0 0
-> 192.168.150.139:0 Route 2 0 0
客户端执行结果：此时客户端访问的443和80端口均会绑定持久在同一台RS上
[root@localhost ~]# curl http://192.168.150.131
<h1>RS1</h1>
[root@localhost ~]# curl -k https://192.168.150.131
<h1>RS1</h1>
原创文章，作者：N23-苏州-void，如若转载，请注明出处：http://www.178linux.com/61846
评论列表（1条）
总结的比较详细，能给出物理拓扑和实验验证的例子会更好~~继续加油~