Linux+nginx+tomcat+JDK整合实现JAVA程序负载均衡

linux资料

本组合使用软件各个版本：

Centos 5   Nginx-0.7.60-linux.tar.gz   tomcat5.5.27 for linux.zip   jdk1_5_0_09-linux.bin   pcre-7.8.tar.gz

操作系统安装不再详细说明，网上相关文档一大堆。

一.软件安装：

安装JDK

将jdk1_5_0_09-linux.bin文件cp到/usr/local目录下

添加权限：chmod u+x jdk1_5_0_09-linux.bin

执行安装：./ jdk1_5_0_09-linux.bin  按照提示输入yes即可完成安装

安装完成后会在/usr/local下有个名为jdk1.5.0_09的目录

由于我做负载均衡，所以安装了两个tomcat，解压tomcat到/usr/local/tomcat1和/usr/local/tomcat2

分别进入/usr/local/tomcat1/bin和/usr/local/tomcat2/bin目录执行命令：chmod u+x *.Sh  设置权限

Tomcat1保持不变，修改tomcat2下conf中server.xml

<Server port="8005" shutdown="SHUTDOWN">

<!-- Define a non-SSL HTTP/1.1 Connector on port 8080 -->

<Connector port="8080" maxHttpHeaderSize="8192"

maxThreads="150" minSpareThreads="25" maxSpareThreads="75"

enableLookups="false" redirectPort="8443" acceptCount="100"

connectionTimeout="20000" disableUploadTimeout="true" />

<!-- Define an AJP 1.3 Connector on port 8009 -->

<Connector port="8009" enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />

将红色字体的端口更改成其他不使用的端口，例如：8005更改为9005，8080更改为8081，8009更改为9009

修改tomcat2下bin中的startup.sh和shutdown.sh添加以下内容

export JAVA_HOME=/usr/local/java1.5.0_09

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=$JAVA_HOME/lib

export CATALINA_HOME=$CATALINA_2_HOME

export CATALINA_BASE=$CATALINA_2_BASE

vi /etc/profile  配置系统变量，打开后添加以下内容：

JAVA_HOME=/usr/local/java1.5.0_09

export JAVA_HOME

CATALINA_BASE=/usr/local/tomcat1

CATALINA_HOME=/usr/local/tomcat1

export CATALINA_BASE CATALINA_HOME

TOMCAT_HOME=/usr/local/tomcat1

export TOMCAT_HOME

CATALINA_2_BASE=/usr/local/tomcat2

CATALINA_2_HOME=/usr/local/tomcat2

export CATALINA_2_BASE CATALINA_2_HOME

TOMCAT_2_HOME=/usr/local/tomcat2

export TOMCAT_2_HOME

JRE_HOME=/usr/local/java1.5.0_09/jre

export JRE_HOME

CLASSPATH=/usr/local/java1.5.0_09/lib: /usr/local/java1.5.0_09/jre/lib

export  CLASSPATH

PATH=$PATH:/usr/local/java1.5.0_09/bin:/usr/local/java1.5.0_09/jre/bin:/usr/local/tomcat1/bin:/usr/local/tomcat2/bin

export PATH

保存退出，logout 注销一下

测试JDK是否安装成功

Java –version

看是否显示为刚刚安装的jdk版本名称

分别进入/usr/local/tomcat1/bin和/usr/local/tomcat2/bin目录执行./startup.sh

访问http://local:8080/和http://localhost:8081/看是否可以看到tomcat的欢迎界面

设置tomcat随系统启动

vi /etc/rc.d/rc.local 添加以下内容

/usr/local/tomcat1/bin/startup.sh

/usr/local/tomcat2/bin/startup.sh

安装nginx，在安装nginx之前需要先安装所需的pcre库

解压我们下载的pcre-7.8.tar.gz

tar zxvf pcre-7.8.tar.gz

cd pcre-7.8

./configure

make

make install

安装完毕

解压nginx-0.7.60-linux.tar.gz

tar zxvf nginx-0.7.60-linux.tar.gz

./configure--user=www--group=www--prefix=/usr/local/nginx--with-http_stub_status_module--with-http_ssl_module

创建代理文件 vi /usr/local/nginx/conf/proxy.conf 输入以下内容

proxy_redirect          off;

proxy_set_header        Host            $host;

proxy_set_header        X-Real-IP       $remote_addr;

proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

client_max_body_size 10m;

client_body_buffer_size 128k;

proxy_connect_timeout 600;

proxy_read_timeout 600;

proxy_send_timeout 600;

proxy_buffer_size 8k;

proxy_buffers 4 32k;

proxy_busy_buffers_size 64k;

proxy_temp_file_write_size 64k;

保存退出。

编辑nginx.conf文件 vi /usr/local/nginx/conf/nginx.conf 修改如下

#运行用户

#user  nobody;

#启用进程

worker_processes  2;

#全局错误日志和pid

error_log  logs/error.log;

#error_log  logs/error.log  notice;

#error_log  logs/error.log  info;

pid        logs/nginx.pid;

#工作模式及连接数上限

worker_rlimit_nofile 51200;

events {

worker_connections  51200;

}

#配置http已经反向代理做负载均衡

http {

include       mime.types;

default_type  application/octet-stream;

include proxy.conf;

#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

#                  '$status $body_bytes_sent "$http_referer" '

#                  '"$http_user_agent" "$http_x_forwarded_for"';

#access_log  logs/access.log  main;

sendfile        on;

#tcp_nopush     on;

#keepalive_timeout  0;

keepalive_timeout  65;

#gzip  on;

#配置负载均衡服务列表,weight参数表示权重值，值越大权重越高

upstream proxyserver {

#ip_hash;

server localhost:8080 weight=6;

server localhost:8081 weight=1;

}

server {

#端口以及域名配置

listen       80;

server_name  www.xxxx.com;

#charset koi8-r;

#access_log  logs/host.access.log  main;

location ~ ^/NginxStatus/ {

stub_status on;

access_log off;

}

#设定默认首页

location / {

root   /wwwroot/ROOT;

index  index.html login.jsp;

}

#禁止访问WEB-INF

location ~ ^/(WEB-INF)/ {

deny all;

}

#设定浏览器访问时nginx直接处理和需缓存的文件后缀以及时间参数

location ~ \.(gif|jpg|jpeg|png|bmp|ico|rar|css|zip|txt|flv|swf|mid|doc|ppt|xls|pdf|txt|mp3|wma)$ {

root /wwwroot/ROOT;

expires 24h;

}

location ~ \.(htm|html)$ {

root /wwwroot/ROOT;

}

#设定此后缀的文件利用反向代理转给负载均衡列表中的tomcat处理

location ~ \.jsp$ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~ \.do$ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~ \.js$ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~ \.xml$ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~ ^/dwr/ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~ ^/html/fckeditor/ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

server {

listen       80;

server_name  www.1234.com;

#charset koi8-r;

#access_log  logs/host.access.log  main;

location ~ ^/NginxStatus/ {

stub_status on;

access_log off;

}

location / {

root   /wwwroot/ROOT;

index  col.html login.jsp;

}

location ~ ^/(WEB-INF)/ {

deny all;

}

location ~ \.(gif|jpg|jpeg|png|bmp|ico|rar|css|zip|txt|flv|swf|mid|doc|ppt|xls|pdf|txt|mp3|wma)$ {

root /wwwroot/ROOT;

expires 24h;

}

location ~ \.(htm|html)$ {

root /wwwroot/ROOT;

}

location ~ \.jsp$ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~ \.do$ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~ \.js$ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~ \.xml$ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~ ^/dwr/ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

location ~ ^/html/fckeditor/ {

proxy_pass http://proxyserver;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

}

}

}

Nginx 启动命令：sbin/nginx

最后就是设置tomcat集群所需的session同步

分别打开tomcat1和tomcat2下conf中server.xml

找到<Engine name="Catalina" defaultHost="localhost">

Tomcat1下的修改为<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker1">

Tomcat2下的修改为<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker2">

找到

<Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"

managerClassName="org.apache.catalina.cluster.session.DeltaManager"

expireSessionsOnShutdown="false"

useDirtyFlag="true"

notifyListenersOnReplication="true">

<Membership

className="org.apache.catalina.cluster.mcast.McastService"

mcastAddr="228.0.0.4"

mcastPort="45564"

mcastFrequency="500"

mcastDropTime="3000"/>

<Receiver

className="org.apache.catalina.cluster.tcp.ReplicationListener"

tcpListenAddress="127.0.0.1"

tcpListenPort="4001"

tcpSelectorTimeout="100"

tcpThreadCount="6"/>

<Sender

className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"

replicationMode="pooled"

ackTimeout="15000"

waitForAck="true"/>

<Valve className="org.apache.catalina.cluster.tcp.ReplicationValve"

filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>

<Deployer className="org.apache.catalina.cluster.deploy.FarmWarDeployer"

tempDir="/tmp/war-temp/"

deployDir="/tmp/war-deploy/"

watchDir="/tmp/war-listen/"

watchEnabled="false"/>

<ClusterListener className="org.apache.catalina.cluster.session.ClusterSessionListener"/>

</Cluster>

将注释去掉启用此功能，注意两处修改

tcpListenAddress="127.0.0.1"

tcpListenPort="4001"

tcpListenAddress修改为此tomcat所在机器的IP地址，如果两个tomcat在同一台服务器上那么tcpListenPort的端口需要修改，如果不在同一台服务器上只需要修改IP即可。

Linux下默认没有开启组播功能，所以需要开下

route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0

如果需要在服务器启动时即开通组播需要修改

Vi /etc/sysconfig/static-routes

添加以下内容

eht0 net 224.0.0.0 netmask 240.0.0.0

还需要在程序的web.xml里面</web-app>前面加入以下这句话

<distributable/>
session同步到这里设置完毕

优化linux内核

vi /etc/sysctl.conf

在末尾添加以下内容

net.ipv4.tcp_fin_timeout = 30

net.ipv4.tcp_keepalive_time = 300

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_tw_recycle = 1

net.ipv4.ip_local_port_range = 5000    65000

使设置立即生效：/sbin/sysctl –p

ulimit –a 查看系统资源限制设定

ulimit -n 4096 修改open file 参数值为4096

用ulimit -n 修改open files 总是不能保持。所以用下面一个简单的办法更好些。
修改/etc/security/limits.conf 添加如下一行：
* - nofile 1006154
修改/etc/pam.d/login添加如下一行
session required /lib/security/pam_limits.so

发表评论