Nginx+keepalived做双机热备加tomcat负载均衡
环境说明：
nginx1:192.168.2.47nginx2:192.168.2.48tomcat1:192.168.2.49tomcat2:192.168.2.50vip:192.168.2.51
一.Nginx配置
1.安装Nginx所需pcre库wgettar -zxvf pcre-8.10.tar.gzcd pcre-8.10./configuremakemake install
2.安装Nginx
wgetgroupadd wwwuseradd -g www wwwtar zxvf nginx-0.8.52.tar.gzcd nginx-0.8.52/./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_modulemakemake install
注：如果出现以下错误
./configure: error: SSL modules require the OpenSSL library.Centos需要安装openssl-develUbuntu则需要安装:sudo apt-get install libssl-dev
3.修改配置文件为以下内容：
user www www;worker_processes 2;pid logs/nginx.pid;worker_rlimit_nofile 51200; events{ use epoll; worker_connections 51200;} http{ include mime.types; default_type application/octet-stream; keepalive_timeout 120; server_tokens off; send_timeout 60; tcp_nodelay on; upstream tomcats { server 192.168.2.50:8080; server 192.168.2.49:8080; #ip_hash; #在没有做共享session的情况下ip_hash可以解决session问题 } server { listen 80; server_name 192.168.2.48; location / { proxy_pass http://tomcats; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } log_format access_log '$remote_addr - $remote_user [$time_local] $request ' '"$status" $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /usr/local/nginx/logs/access.log access_log; } } 4.测试配置文件
/usr/local/nginx/sbin/nginx -t
如果出现以下情况
/usr/local/nginx/sbin/nginx: error while loading shared libraries: libpcre.so.0: or directory
解决方法：
sudo ln -s /usr/local/lib/libpcre.so.0 /usr/lib/libpcre.so.0
/usr/local/nginx/sbin/nginx -t
显示以下信息为正确的the configuration file /usr/local/nginx/conf/nginx.conf syntax is okconfiguration file /usr/local/nginx/conf/nginx.conf test is successful
5.优化内核参数
vim /etc/sysctl.conf在最后添加net.ipv4.tcp_max_syn_backlog = 65536net.core.netdev_max_backlog = 32768net.core.somaxconn = 32768net.core.wmem_default = 8388608net.core.rmem_default = 8388608net.core.rmem_max = 16777216net.core.wmem_max = 16777216net.ipv4.tcp_timestamps = 0net.ipv4.tcp_synack_retries = 2net.ipv4.tcp_syn_retries = 2net.ipv4.tcp_tw_recycle = 1net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_mem = 94500000 915000000 927000000net.ipv4.tcp_max_orphans = 3276800net.ipv4.ip_local_port_range = 1024 65535
保存退出后执行
sysctl -p
6.切割Nginx日志脚本
#!/bin/bashPATH_LOGS="/usr/local/nginx/logs"YEAR=`date -d "-1 days" +"%Y"`MONTH=`date -d "-1 days" +"%m"`mkdir -p $PATH_LOGS/$YEAR/$MONTHmv $PATH_LOGS/access.log $PATH_LOGS/$YEAR/$MONTH/access_$(date -d "-1 days" +"%Y%m%d").logkill -USR1 `cat $PATH_LOGS/nginx.pid`
把该脚本加到crontab每天00点执行
注：备机的Nginx和以上安装步骤一样二.安装配置Keepalived
1.下载所需要的软件wgetwget 2.安装popt编译keepalived时需要popt，否则会报以下错误：configure: error: Popt libraries is required
tar -zxvf popt-1.16.tar.gzcd popt-1.16./configuremakemake install
3.安装keepalived
tar -zxvf keepalived-1.1.19.tar.gzcd keepalived-1.1.19./configure --prefix=/usr/local/keepalivedmakemake install
4.修改配置文件为以下内容:
vim /usr/local/keepalived/etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { router_id LVS_DEVEL}vrrp_script Monitor_Nginx { script "/root/scripts/monitor_nginx.sh" #根据自己的实际路径放置monitor_nginx.sh interval 2 weight 2}vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1234} track_script { Monitor_Nginx} virtual_ipaddress { 192.168.2.51 }} 注：monitor_nginx.sh为监控nginx进程的脚本,内容如下
#!/bin/bashif [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]then /usr/local/nginx/sbin/nginx sleep 5 if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ] then killall keepalived fifi
5.启动keepalived
/usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf
注:备机的keepalived的安装和上面一样，只要把配置文件改为以下(把MASTER改为BACKUP)
! Configuration File for keepalived global_defs { router_id LVS_DEVEL}vrrp_script Monitor_Nginx { script "/root/scripts/monitor_nginx.sh" interval 2 weight 2}vrrp_instance VI_1 { state BACKUP #改为BACKUP interface eth0 virtual_router_id 51 priority 100 #比MASTER数值要低 advert_int 1 authentication { auth_type PASS auth_pass 1234} track_script { Monitor_Nginx} virtual_ipaddress { 192.168.2.51 }} 三．测试步骤
1． 访问VIP看是否能够正常访问后端的tomcat
2． 停止其中一个tomcat看是否能将访问转到另一台上3． 停止两台nginx上任何一个nginx进程看监控进程脚本是否会自动启动nginx4． 停止任何一台nginx上的keepalived进程看另一台是否接管vip比如停止Master上的keepalived,例如如下killall keepalived,查看BACKUP机器是否已经接管，如果BACKUP接管后，BACKUP机器日志会是出下情况tail /var/log/syslogKeepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATEKeepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATEKeepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.51
MASTER机器上日志会显示
Keepalived_vrrp: Terminating VRRP child process on signalKeepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
现在把MASTER上的Keepalived重新启动，会看到MASTER重新接管VIP，并对外提供服务，BACKUP仍旧回到BACKUP STATE,如果不是这种情况，请检查配置文件和步骤.
现在的BACKUP日志如下：
Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advertKeepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATEKeepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Master日志如下：
Keepalived_vrrp: VRRP_Script(Monitor_Nginx) succeededKeepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATEKeepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATEKeepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.51 ------------------------------------------------------
nginx可以根据客户端IP进行负载均衡，在upstream里设置ip_hash，就可以针对同一个C类地址段中的客户端选择同一个后端服务器，除非那个后端服务器宕了才会换一个。
nginx的upstream目前支持的5种方式的分配
1、轮询（默认） 每个请求按时间顺序逐一分配到不同的后端服务器，如果后端服务器down掉，能自动剔除。 upstream backserver { server 192.168.0.14; server 192.168.0.15; } 2、指定权重 指定轮询几率，weight和访问比率成正比，用于后端服务器性能不均的情况。 upstream backserver { server 192.168.0.14 weight=10; server 192.168.0.15 weight=10; } 3、IP绑定 ip_hash 每个请求按访问ip的hash结果分配，这样每个访客固定访问一个后端服务器，可以解决session的问题。 upstream backserver { ip_hash; server 192.168.0.14:88; server 192.168.0.15:80; } 4、fair（第三方） 按后端服务器的响应时间来分配请求，响应时间短的优先分配。 upstream backserver { server server1; server server2; fair; } 5、url_hash（第三方） 按访问url的hash结果来分配请求，使每个url定向到同一个后端服务器，后端服务器为缓存时比较有效。 upstream backserver { server squid1:3128; server squid2:3128; hash $request_uri; hash_method crc32; } 在需要使用负载均衡的server中增加 proxy_pass http://backserver/; upstream backserver{ ip_hash; server 127.0.0.1:9090 down; (down 表示单前的server暂时不参与负载) server 127.0.0.1:8080 weight=2; (weight 默认为1.weight越大，负载的权重就越大) server 127.0.0.1:6060; server 127.0.0.1:7070 backup; (其它所有的非backup机器down或者忙的时候，请求backup机器) } max_fails ：允许请求失败的次数默认为1.当超过最大次数时，返回proxy_next_upstream 模块定义的错误 fail_timeout:max_fails次失败后，暂停的时间