当前位置:首页 > 技术文章 > openx > 基于Nginx, lighttpd/apache上的OpenX高性能集群配置手册
基于Nginx, lighttpd/apache上的OpenX高性能集群配置手册
文章来源:本站原创  浏览次数:300  发布日期:2013-04-16

本文英文原文地址:http://www.sherin.co.in/openxcluster121/

编者注:本文非常详细的描述了OpenX集群的配置方法,对于想通过OpenX做大规模服务的应用来说应该有比较大的帮助。原文是英文的,为了保证意思的表达就不翻译成中文了。

1. Introduction

1.1 Why the name 121 Cluster?

1.2 Technology

2. System Requirements

2.1 Load Balancer Node
2.2 Web Servers Node
2.3 Database Server Node

3. Install and Configure Load Balancer4. Install and Configure Webserver

4.1 File system configuration
4.2 Lighttpd installation
4.3 Apache installation

5. Install and Configure Database Server6. OptimizationAppendix-A About The AuthorAppendix-B License1. Introduction

This document help you to setup a simple cluster for starters. This configuration using the best and popular software configuration for getting better stability and performance. This can be used for Openx cluster platforms and large vbulletin forums. Please not this is not a high availability solution but cluster. You can build this cluster with a minimum of 4 servers.

1.1 Why the name 121 Cluster?

Let me explain 1-2-1 cluster means ,

  • 1 – Load balancer

  • 2 – Werbserver , or multiple of 2

  • 1 – Database server

So this configuration is based on one load balancer , two or multiple of 2’s of webservers and one database server. Here all are servers there is no need to buy external devices. This is the simplest and well optimized cluster that you can build. You can see a graphical representation of this cluster as below,

1.2 Technology

We are here using Nginx webserver as http load balanced Proxy. It is the most popular proxy application and around 6% of high traffic websites in this world usin nginx. Some example sites using nginx are wordpress.com ,rambler.ru,fastmail.fm

Behind these proxies we use lighttpd as webservers for hosting our application. Lighttpd is a fast webserver with light foot print. It can simply handle millions of requests without increasing server load.It is designed and optimized for high performance environments. With a small memory footprint compared to other web-servers, effective management of the cpu-load, and advanced feature set (FastCGI, SCGI, Auth, Output-Compression, URL-Rewriting and many more) lighttpd is the perfect solution for every server that is suffering load problems.

2. System Requirements

Here we using the best webservers and proxy applications available on industry. It is fully a service based cluster. So no worry about kernel panics and OS corruptions. Also if you have a private LAN across these nodes, it is good to configure it otherwise you need to proxify the access over internet.

2.1 Load Balancer Node

This node is your public node,, That means your Openx/website domain is pointing to this server IP. I recommend to install ubuntu server Operating system/ Centos for your load balanced server. Here in this article I am mentioning the procedure based on ubuntu.

2.2 Web Servers Node

You may need a minimum of two webservers or multiple of 2, beacuse I am configuring GlusterFS between these webservers for file sharing. GlusterFS is one of the top cluster file system which is built on ext3. With 4 webservers you can configure a RAID10 like GlusterFs file system. That mean your cluster will work if 50% of node go down. It have automatic mirroring and scaling capacity. So there is no need to sync your website / openx contents regularly.

You can chose centos/ubuntu Os as operating system in webserver nodes. If you going to install cPanel then you can only use apache as webservers. Also remember the webservers must need same type of hardware configurations.

2.3 Database Server Node

As it is a single cluster we only use one database sever for Mysql. I recommend to use a bigger configuration of for this server as follows,

  • Processor -Dual Quad core AMD/ Xeon

  • RAM – 12 GB

  • HDD – SAS RAID 10

Again Use Centos as operating system in this server

3. Install and Configure Load Balancer

Please see the picture , the top one is load balance. Let us use the same IP as in this example. Download the latest stable version of Nginx from here .I used nginx-0.7.62.tar.gz . Please proceed as follows.

# wget -c http://sysoev.ru/nginx/nginx-0.7.62.tar.gz# tar -xzf nginx-0.7.62.tar.gz# cd nginx-0.7.62/# ./configure --prefix=/opt/nginx# make# make install

Please resolve the dependencies before make.

Now we installed Nginx here. It is a best webserver eventhough we can use it as load balanced proxy. The Nginx configuration file is /opt/nginx/conf/nginx.conf . Add the following contents to /opt/nginx/conf/nginx.conf

user  nobody;worker_processes  5;error_log  logs/error.log;pid        logs/nginx.pid;events {    worker_connections  4096;}http {    include     mime.types;    default_type  application/octet-stream;    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '                      '$status $body_bytes_sent "$http_referer" '                      '"$http_user_agent" "$http_x_forwarded_for"';    access_log  logs/access.log  main;    sendfile        on;    tcp_nopush     on;    keepalive_timeout  65;    #gzip  on;        upstream my_openx_site {        server 10.0.0.11:80 weight=5;  # this is your webserver www1        server 10.0.0.12:80 weight=5;  # this is your web server www2        }        server {                 listen 80;                server_name localhost;                location / {                         proxy_pass http://my_openx_site;                }        }}

The above configuration have 5 worker processor and can handle 4096 requests per second. If you need more connections increase the limits.
Now add a user and group

# useradd nobody# groupadd nobody

Now give suitable permission for log folders.

# chown -R nobody.nobody  /opt/nginx/logs/

Now we need to create a startup script. I have done some modification for the initscript as follows. Remember this is for ubuntu. Please copy the following contents to /etc/init.d/nginx

#! /bin/sh### BEGIN INIT INFO# Provides:          nginx# Required-Start:    $remote_fs $syslog# Required-Stop:     $remote_fs $syslog# Default-Start:     2 3 4 5# Default-Stop:      0 1 6# Short-Description: nginx init.d script for Ubuntu 8.10 and lesser versions.# Description:       nginx init.d script for Ubuntu 8.10 and lesser versions.### END INIT INFO#------------------------------------------------------------------------------# nginx - this script, which starts and stops the nginx daemon for ubuntu.## description:  Nginx is an HTTP(S) server, HTTP(S) reverse \#               proxy and IMAP/POP3 proxy server.  This \#               script will manage the initiation of the \#               server and its process state.## processname: nginx# config:      /opt/nginx/conf/nginx.conf# pidfile:     /opt/nginx/logs/nginx.pid# Provides:    nginx#                                                                                                                                                                                                                                 # Notes: nginx init.d script for Ubuntu 8.10 and lesser versions. Functions#------------------------------------------------------------------------------. /lib/lsb/init-functions                                                                                                                                                                                                         #------------------------------------------------------------------------------#                               Consts#------------------------------------------------------------------------------PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/binDAEMON=/opt/nginx/sbin/nginx                                                                                                                                                                                                      NAME=nginxDESCRIPTION="Nginx Server..."                                                                                                                                                                                                     PIDSPATH=/opt/nginx/logsPS=$NAME                                #the process, which happens to be the NAMEPIDFILE=$NAME.pid                       #pid fileRUNAS=root                              #user to run as                                                                                                                                                                           SCRIPT_OK=0                             #ala error codesSCRIPT_ERROR=1                          #ala error codesTRUE=1                                  #booleanFALSE=0                                 #boolean                                                                                                                                                                                  lockfile=/var/lock/subsys/nginxNGINX_CONF_FILE="/opt/nginx/conf/nginx.conf"                                                                                                                                                                                      #------------------------------------------------------------------------------#                               Simple Tests#------------------------------------------------------------------------------                                                                                                                                                   #test if nginx is a file and executabletest -x $DAEMON || exit 0                                                                                                                                                                                                         # Include nginx defaults if availableif [ -f /etc/default/nginx ] ; then        . /etc/default/nginxfi                                                                                                                                                                                                                                #set exit condition#set -e                                                                                                                                                                                                                           #------------------------------------------------------------------------------#                               Functions#------------------------------------------------------------------------------                                                                                                                                                   configtest() {        $DAEMON -t -c $NGINX_CONF_FILE}                                                                                                                                                                                                                                 getPSCount() {        return `pgrep -f $PS | wc -l`}                                                                                                                                                                                                                                 isRunning(){        pidof_daemon        PID=$?                                                                                                                                                                                                                            if [ $PID -gt 0 ]; then                return 1        else                return 0        fi}                                                                                                                                                                                                                                 status(){        isRunning        isAlive=$?                                                                                                                                                                                                                        if [ "${isAlive}" -eq $TRUE ]; then                echo "$NAME found running with processes:  `pidof $PS`"        else                echo "$NAME is NOT running."        fi                                                                                                                                                                                                                        }                                                                                                                                                                                                                                 removePIDFile(){        if [ -f $PIDSPATH/$NAME.pid ]; then                rm $PIDSPATH/$NAME.pid        fi}                                                                                                                                                                                                                                 start() {        log_daemon_msg "Starting $DESCRIPTION"                                                                                                                                                                                            isRunning        isAlive=$?                                                                                                                                                                                                                        if [ "${isAlive}" -eq $TRUE ]; then                log_end_msg $SCRIPT_ERROR        else                start-stop-daemon --start --quiet --chuid $RUNAS --pidfile $PIDSPATH/$PIDFILE --exec $DAEMON                chmod 400 $PIDSPATH/$PIDFILE                log_end_msg $SCRIPT_OK        fi}                                                                                                                                                                                                                                 stop() {        log_daemon_msg "Stopping $DESCRIPTION"                                                                                                                                                                                            isRunning        isAlive=$?        if [ "${isAlive}" -eq $TRUE ]; then                start-stop-daemon --stop --quiet --pidfile $PIDSPATH/$PIDFILE                                                                                                                                                                     removePIDFile                                                                                                                                                                                                                     log_end_msg $SCRIPT_OK        else                log_end_msg $SCRIPT_ERROR        fi}                                                                                                                                                                                                                                 reload() {        configtest || return $?                                                                                                                                                                                                           log_daemon_msg "Reloading (via HUP) $DESCRIPTION"                                                                                                                                                                                 isRunning        if [ $? -eq $TRUE ]; then                `killall -HUP $PS` #to be safe                                                                                                                                                                                                    log_end_msg $SCRIPT_OK        else                log_end_msg $SCRIPT_ERROR        fi}                                                                                                                                                                                                                                 terminate() {        log_daemon_msg "Force terminating (via KILL) $DESCRIPTION"                                                                                                                                                                        PIDS=`pidof $PS` || true                                                                                                                                                                                                          [ -e $PIDSPATH/$PIDFILE ] && PIDS2=`cat $PIDSPATH/$PIDFILE`                                                                                                                                                                       for i in $PIDS; do                if [ "$i" = "$PIDS2" ]; then                        kill $i                        removePIDFile                fi        done                                                                                                                                                                                                                              log_end_msg $SCRIPT_OK                                                                                                                                                                                                    }pidof_daemon() {    PIDS=`pidof $PS` || true    [ -e $PIDSPATH/$PIDFILE ] && PIDS2=`cat $PIDSPATH/$PIDFILE`    for i in $PIDS; do        if [ "$i" = "$PIDS2" ]; then            return 1        fi    done    return 0}case "$1" in  start)        start        ;;  stop)        stop        ;;  restart|force-reload)        stop        start        ;;  reload)        $1        ;;  status)        status        ;;  configtest)        $1        ;;  terminate)        $1        ;;  *)        FULLPATH=/etc/init.d/$NAME        echo "Usage: $FULLPATH {start|stop|restart|force-reload|status|configtest|terminate}"        exit 1        ;;esacexit 0

Now give execute permission to this script

# chmod 755 /etc/init.d/nginx

This is all about Load balancer configuration. You can start the Load balance, before that you may need to build your webservers as described below,

4. Install and Configure Webserver

Now we need to configure our web servers. All request came to Load balancer will be passed to your webserver. You can use lighttpd or apache2.2.x as webserver . I recommend to you lighttpd as webserver , because it can handle very high traffic websites with zero load. Before that we are going to configure gluster fs file system among the webserver. These file system is very scalable and high available.

4.1 File system configuration

If you don’t need a common file system, please skip this step.Let us use the minimal number of webservers for Glusterfs. If you have 4 webservers, it is easy to setup a RAID10 model cluster file system. Here we use two servers.
Download the latest stable version of glusterfs from http://www.gluster.com/ . I here used version glusterfs-2.0.2

# tar -xzf   glusterfs-2.0.2.tar.gz# cd  glusterfs-2.0.2/# ./configure# make# make install

Now create the configuration files as follows.

# mkdir /etc/glusterfs/# touch /etc/glusterfs/glusterfsd.vol

The server itself can act as server and client . The GFS server configuration is /etc/glusterfs/glusterfsd.vol . Now add the following contents to this file.

# file: /etc/glusterfs/glusterfsd.volvolume posix  type storage/posix  option directory /data    # this partition is sharingend-volumevolume locks  type features/locks  subvolumes posixend-volumevolume brick  type performance/io-threads  option thread-count 8  subvolumes locksend-volumevolume server  type protocol/server  option transport-type tcp  option auth.addr.brick.allow *  subvolumes brickend-volume

Now create the GlusterFS client configuration file /etc/glusterfs/glusterfs-client.vol as follows

# file  /etc/glusterfs/glusterfs-client.volvolume remote1  type protocol/client  option transport-type tcp  option remote-host 10.0.0.11  option remote-subvolume brickend-volumevolume remote2  type protocol/client  option transport-type tcp  option remote-host 10.0.0.12  option remote-subvolume brickend-volumevolume distribute  type cluster/distribute  subvolumes remote1 remote2end-volumevolume writebehind  type performance/write-behind  option window-size 1MB  subvolumes distributeend-volumevolume cache  type performance/io-cache  option cache-size 512MB subvolumes writebehindend-volum

So now we have both client and server configuration for the Glusterfs file system. Here the /data partition is sharing around the webservers. You may need to create same configuration in all webservers.
Now start the glusterfs server

# /etc/init.d/glusterfsd  start

Now mount the file system to /home as follows

# glusterfs -f /etc/glusterfs/glusterfs-client.vol /home

Add the above line to rc.local file so during reboot your file system will mount automatically. So now we have a common files system among the webserver nodes.

4.2 Lighttpd installation

Please note either you install lighttpd as webserver or use apache. If you going to chose apache please jump to next step.
Download the lighty from http://www.lighttpd.net/download I used the latest version.

# wget -c http://www.lighttpd.net/download/lighttpd-1.4.23.tar.gz# tar -xzf lighttpd-1.4.23.tar.gz# cd lighttpd-1.4.23/# ./configure --prefix=/opt/lighttpd --enable-fastcgi  --with-openssl# make# make install# mkdir -p /etc/lighttpd/# cp doc/lighttpd.conf  /etc/lighttpd/# cp doc/rc.lighttpd.redhat  /etc/init.d/lighttpd# chmod 755  /etc/init.d/lighttpd

Now edit the init script /etc/init.d/lighttpd and update the following line as given below,

lighttpd="/opt/lighttpd/sbin/lighttpd"

ow we need to configure lighttpd with php. Please install php as fcgi module. It is easy
Download the latest stable php from http://php.net/ and extract it. Now build it as follows,

# /configure --prefix=/usr/local/php5/  \ --with-config-file-path=/usr/local/php5/etc \  --enable-force-cgi-redirect --enable-fastcgi\ --with-gd --with-jpeg-dir=/usr/local --with-zlib \ --with-openssl  --with-mysql# make# make install# cp php.ini-dist  /usr/local/php5/etc/php.ini

Now edit /etc/lighttpd/lighttpd.conf as follows.

server.modules              = (                               "mod_access",                               "mod_fastcgi",                               "mod_proxy",                               "mod_scgi",                               "mod_simple_vhost",                               "mod_evhost",                               "mod_accesslog" ) server.document-root        = "/home/openx/public_html"server.errorlog             = "/var/log/lighttpd/error.log"index-file.names            = ( "index.php", "index.html",                               "index.htm", "default.htm" )mimetype.assign             = (  ".pdf"          =>      "application/pdf",  ".sig"          =>      "application/pgp-signature",  ".spl"          =>      "application/futuresplash",  ".class"        =>      "application/octet-stream",  ".ps"           =>      "application/postscript",  ".torrent"      =>      "application/x-bittorrent",  ".dvi"          =>      "application/x-dvi",  ".gz"           =>      "application/x-gzip",  ".pac"          =>      "application/x-ns-proxy-autoconfig",  ".swf"          =>      "application/x-shockwave-flash",  ".tar.gz"       =>      "application/x-tgz",  ".tgz"          =>      "application/x-tgz",  ".tar"          =>      "application/x-tar",  ".zip"          =>      "application/zip",  ".mp3"          =>      "audio/mpeg",  ".m3u"          =>      "audio/x-mpegurl",  ".wma"          =>      "audio/x-ms-wma",  ".wax"          =>      "audio/x-ms-wax",  ".ogg"          =>      "application/ogg",  ".wav"          =>      "audio/x-wav",  ".gif"          =>      "image/gif",  ".jar"          =>      "application/x-java-archive",  ".jpg"          =>      "image/jpeg",  ".jpeg"         =>      "image/jpeg",  ".png"          =>      "image/png",  ".xbm"          =>      "image/x-xbitmap",  ".xpm"          =>      "image/x-xpixmap",  ".xwd"          =>      "image/x-xwindowdump",  ".css"          =>      "text/css",  ".html"         =>      "text/html",  ".htm"          =>      "text/html",  ".js"           =>      "text/javascript",  ".asc"          =>      "text/plain",  ".c"            =>      "text/plain",  ".cpp"          =>      "text/plain",  ".log"          =>      "text/plain",  ".conf"         =>      "text/plain",  ".text"         =>      "text/plain",  ".txt"          =>      "text/plain",  ".dtd"          =>      "text/xml",  ".xml"          =>      "text/xml",  ".mpeg"         =>      "video/mpeg",  ".mpg"          =>      "video/mpeg",  ".mov"          =>      "video/quicktime",  ".qt"           =>      "video/quicktime",  ".avi"          =>      "video/x-msvideo",  ".asf"          =>      "video/x-ms-asf",  ".asx"          =>      "video/x-ms-asf",  ".wmv"          =>      "video/x-ms-wmv",  ".bz2"          =>      "application/x-bzip",  ".tbz"          =>      "application/x-bzip-compressed-tar",  ".tar.bz2"      =>      "application/x-bzip-compressed-tar",  ""              =>      "application/octet-stream", )                                                            accesslog.filename          = "/var/log/lighttpd/access.log"url.access-deny             = ( "~", ".inc" )$HTTP["url"] =~ "\.pdf$" {  server.range-requests = "disable"}static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )fastcgi.server = ( ".php" => ((                     "bin-path" => "/usr/local/php5/bin/php",                     "socket" => "/tmp/php.socket",                     "max-procs" => 2,                     "bin-environment" => (                       "PHP_FCGI_CHILDREN" => "16",                       "PHP_FCGI_MAX_REQUESTS" => "10000"                     ),                     "bin-copy-environment" => (                       "PATH", "SHELL", "USER"                     ),                     "broken-scriptfilename" => "enable"                 )))

So the above configuration will handle 10000 requests per children per second. You can adjust the factcgi.server options as you wish. Remember it is good all your webservers have same type configuration.

4.3 Apache installation

if you have cpanel or you are going to use apache2.2.x as webserver, please read my previous article to know how to configure it. Openx Hand Book

5. Install and Configure Database Server

Now you can install your mysql server and configure it.

# yum install mysql-server -y

Edit the my.cnf as follows(Remember it depend on your hardware )

[mysqld]safe-show-databaseold-passwords = 1max_connections =2048max_user_connections = 1024key_buffer_size = 2048Mmyisam_sort_buffer_size = 64Mjoin_buffer_size = 1Mread_buffer_size = 1Msort_buffer_size = 2Mtable_cache = 4000thread_cache_size = 384wait_timeout = 20connect_timeout = 10tmp_table_size = 2048Mmax_heap_table_size = 512Mmax_allowed_packet = 64Mnet_buffer_length = 16384max_connect_errors = 10thread_concurrency = 16concurrent_insert = 2table_lock_wait_timeout = 30read_rnd_buffer_size = 786432bulk_insert_buffer_size = 8Mquery_cache_limit = 7Mquery_cache_size = 64Mquery_cache_type = 1query_prealloc_size = 262144query_alloc_block_size = 65536transaction_alloc_block_size = 8192transaction_prealloc_size = 4096max_write_lock_count = 16long_query_time = 5skip-name-resolveskip-locking[mysqld_safe]open_files_limit = 8192[mysqldump]quickmax_allowed_packet = 16M[myisamchk]key_buffer = 384Msort_buffer = 384Mread_buffer = 256Mwrite_buffer = 256M

Now restart mysql server and create a database as follows

# mysqladmin create openx_db

Now go to mysql command prompt and create a user and password as follows

#mysql>grant all privileges on openx_db.* to 'openx_user'@'%' identified by 'VERYSTRONGPASSWORD';

So now for your openx configuration use host as your mysql serve IP(here 10.0.0.20) and database user and password same as above.

6. Optimization

Now it is time to optimize your servers. Please read my Openx Handbook for optimizations ,

Appendix-A About The Author

My Name : Sherin A
My Web Site : http://www.sherin.co.in/ If you wan’t my designation , you can call me, System Engineer , Security Expert , Software Engineer or a FOSS member.
About me :
1) Founder of vcPanel ( Virtual Private Server Control panel )
2) Founder of FOSS award winner project ffmpegistaller )
3) Founder of FossBlog ( A Free and Open Source Software Blog
My relation with Adds : I am here with adds since 2003. Now providing Openx services for companies US , Canad, UK, Australia, Japan , India ,Germany, Croatia and Russia. Contact Me : All my contacts are available here or send email to me sherinmon[at]gmail[dot]com

Appendix-B License

This article is licensed under Creative Commons Attribution-Noncommercial 2.5 India It means you can share and redistribute it , but give a link back to this original document, because sometimes I will add more tips to this document.


原文来自:基于Nginx, lighttpd/apache上的OpenX高性能集群配置手册 http://www.redyun.net/technology/94.html

红云案例Case