Apache+Tomcat+Mysql load balancing and clustering

Recently developed a Web site, it is estimated that the maximum online 30000, up to 100 concurrent. Development site can stand the pressure, how to ensure that there is no problem loading the site, after decisions are as follows:

1. The use of load balancing and clustering
2. Stress-testing tool Loadrunner

Hardware devices:

In order to be able to carry out stress tests need to set up an environment. Beginning to test in the company LAN, but soon discovered a problem, that is, the pressure of a test script is different every time, and vary widely. The original network is affected by the impact of the company decided to build a completely isolated test network. After the configuration of the LAN structure is as follows:

1. Network speed: 100M
2. Three servers:
   Load Balancing Server: Windows 2003
   Tomcat Server: Windows 2000
   Database server:Windows 2000
   CPU :P4 2.4G
   Memory:1G

Software Version:

Apache ：2.054，
Tomcat:5.0.30,
mysql ：4.1.14.
JDK1.5
Loadrunner7.8

Load balancing program:

A machine to install apache, as a load balancing server, and install tomcat as a worker; a separate installation of tomcat, as the second worker; as a database server.
Apache and tomcat load balancing using JK1.2.14.

Cluster:
Tomcat clusters using the program itself. Configuration in server.xml.

Stress-testing issues:
Stress testing and found some problems, as follows:

1. Tomcat clusters used, the reaction rate becomes very slow. Mainly due to session replication.The role of replication is mainly used to fault-tolerant, that is, there is a machine failure, apache request can be automatically forwarded to another machine. And speed in fault-tolerant considerations, we opted to speed, to remove the Tomcat cluster.
2. OS limits the maximum concurrent users:
   In order to test the pressure of the site, we only test the maximum load the number of Tomcat. Tomcat server OS is  windows2000. When we use stress-testing tool, concurrent test, it was found that as long as more than 15 concurrent users will often not be able to connect server. After some research and found that the problem is the OS: windows2000 support concurrent access to a user is limited, the default 15. So we all use windows 2003 server version.
3. Database Connection Pool:
   Testing database connectivity, it was found that the database connection is slow. Each additional number of users, a lot of poor connectivity. We use the database connection pool is to DBCP, the default initialization for the 50, it should not be very slow. Query the database of connections was found to initialize, initialize a connection only. When an additional concurrent users, the program will re-create a connection, resulting in very slow connections. The reason here. How do we solve this? Occasionally under JDK1.4 implementation of database connection Tomcat5.0.30 stress tests and found that fast, the procedure to create database connection speed is fast. So we modified version 1.4 of the JDK.
4. C3P0 and DBCP
   C3P0 is the default Hibernate3.0 own database connection pool, DBCP was developed by Apache Database Connection Pool. We have two types of connection pool contrast to stress tests and found that 300 concurrent users in the following hours, DBCP faster than the average time C3P0 about 1 seconds. However, in 400 concurrent users, almost two.
   Although the speed of C3P0 faster than DBCP, but BUG: When DBCP database connection established, after the cut off for one reason or another, DBCP will not re-create a new connection, resulting in the need to restart Tomcat in order to solve the problem. BUG of DBCP that we decided to use the database as a C3P0 connection pool.

The revised program:

OS:Windows2003 server version
JDK1.4
Tomcat 5.0.30
Database Connection Pool C3P0
Only the use of load balancing, do not use the cluster.

Software configuration:

Apache configuration:
main configuration httpd.conf file and the new workers.properties
Httpd.conf:
# A request to connect the largest number of
MaxKeepAliveRequests 10000
# NT environment, this parameter can only be configured to provide performance
<IfModule Mpm_winnt.c>
# Of threads per process, maximum 1920. NT only father and son to start two processes can not be set up to start a number of process
ThreadsPerChild 1900
Each child process to handle requests for the largest number of
MaxRequestsPerChild 10000
</IfModule>
# Load mod_jk
#
LoadModule jk_module modules/mod_jk.so
#
# Configure mod_jk
#
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel info
# The request of the distribution of jsp files,. Do dynamic processing the request by tomcat
DocumentRoot “C: /Apache/htdocs”
JkMount /*. jsp loadbalancer
JkMount /*. do loadbalancer
JkMount /servlet/* loadbalancer
# Turn off the host Lookup, if it is on, it affected performance, can have more than 10 seconds delay.
HostnameLookups Off
# Cache configuration
LoadModule cache_module modules/mod_cache.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule mem_cache_module modules/mod_mem_cache.so

<IfModule Mod_cache.c>
CacheForceCompletion 100
CacheDefaultExpire 3600
CacheMaxExpire 86400
CacheLastModifiedFactor 0.1

<IfModule Mod_disk_cache.c>
CacheEnable disk /
CacheRoot c: /cacheroot
CacheSize 327680
CacheDirLength 4
CacheDirLevels 5
CacheGcInterval 4
</IfModule>
<IfModule Mod_mem_cache.c>
CacheEnable mem /
MCacheSize 8192
MCacheMaxObjectCount 10000
MCacheMinObjectSize 1
MCacheMaxObjectSize 51200
</IfModule>
</IfModule>

worker. Properties file
#
# Workers.properties, can refer to
http://jakarta.apache.org/tomcat … config / workers.html
# In Unix, we use forward slashes:
ps =

# List the workers by name

worker.list = tomcat1, tomcat2, loadbalancer

# ————————
# First tomcat server
# ————————
worker.tomcat1.port = 8009
worker.tomcat1.host = localhost
worker.tomcat1.type = ajp13

# Specify the size of the open connection cache.
# worker.tomcat1.cachesize

#
# Specifies the load balance factor when used with
# A load balancing worker.
# Note:
# —-> Lbfactor must be> 0
# —-> Low lbfactor means less work done by the worker.
worker.tomcat1.lbfactor = 900

# ————————
# Second tomcat server
# ————————
worker.tomcat1.port = 8009
worker.tomcat1.host = 202.88.8.101
worker.tomcat1.type = ajp13

# Specify the size of the open connection cache.
# worker.tomcat1.cachesize

#
# Specifies the load balance factor when used with
# A load balancing worker.
# Note:
# —-> Lbfactor must be> 0
# —-> Low lbfactor means less work done by the worker.
worker.tomcat1.lbfactor = 2000
# ————————
# Load Balancer worker
# ————————

#
# The loadbalancer (type lb) worker performs weighted round-robin
# Load balancing with sticky sessions.
# Note:
# —-> If a worker dies, the load balancer will check its state
# Once in a while. Until then all work is redirected to peer
# Worker.
worker.loadbalancer.type = lb
worker.loadbalancer.balanced_workers = tomcat1, tomcat2

#
# END workers.properties
#

Tomcat1 configurations:
<! – Configure server.xml
Remove the 8080 port, that is, comment out the following code: ->
<Connector
port = “8080″ maxThreads = “150″ minSpareThreads = “25″ maxSpareThreads = “75″
enableLookups = “false” redirectPort = “8443″ acceptCount = “100″
debug = “0″ connectionTimeout = “20000″
disableUploadTimeout = “true” />

<! – 8009 port configuration is as follows: ->
<Connector port = “8009″
maxThreads = “500″ minSpareThreads = “400″ maxSpareThreads = “450″
enableLookups = “false” redirectPort = “8443″ debug = “0″
protocol = “AJP/1.3″ />
<! – Configuration Engine ->
<Engine Name=”Catalina” defaultHost=”localhost” debug=”0″ jvmRoute=”tomcat1″>

Start the memory configuration, the development of procedures to configure tomcat configuration:
Initial memory pool: 200 M
Maxinum memory pool: 300M
Tomcat2 configurations:
Almost tomcat1 configuration and the need for changes in the following areas:
<! – Configuration Engine ->
<Engine Name=”Catalina” defaultHost=”localhost” debug=”0″ jvmRoute=”tomcat2″>

Start the memory configuration, the development of procedures to configure tomcat configuration:
Initial memory pool: 512 M
Maxinum memory pool: 768M
Mysql configuration:
Server Type: Dedicated MySQL Server Machine
Database usage: Transational Database Only
The number of concurrent connections: Online Transaction Processing (OLTP)
Character set: UTF8
Database connection pool configuration:
We use the spring framework, configuration is as follows:

<property name=”hibernateProperties”>
<props>
<prop key=”hibernate.dialect”> org.hibernate.dialect.MySQLDialect </ prop>
<prop key=”hibernate.connection.driver_class”> com.mysql.jdbc.Driver </ prop>
<prop key=”hibernate.connection.url”> jdbc: mysql: //www.cpworld2000.com/db </ prop>
<prop key=”hibernate.connection.username”> root</ prop>
<prop key=”hibernate.connection.password”>root</ prop>

<prop key=”hibernate.show_sql”> false </ prop>
<prop key=”hibernate.use_sql_comments”> false </ prop>

<prop key=”hibernate.cglib.use_reflection_optimizer”> true </ prop>
<prop key=”hibernate.max_fetch_depth”> 2 </ prop>

<prop key=”hibernate.c3p0.max_size”> 200 </ prop>
<prop key=”hibernate.c3p0.min_size”> 5 </ prop>
<prop key=”hibernate.c3p0.timeout”> 12000 </ prop>
<prop key=”hibernate.c3p0.max_statements”> 50 </ prop>
<prop key=”hibernate.c3p0.acquire_increment”> 1 </ prop>
</ props>
</ property>