Highly Available Web Pages, Apache and PHP as an Example

Standard

Several weeks ago, I discussed how to have a highly available file storage and a highly available relational database.  With a robust supply of file system and database service, we can stack more services on top.  Today, I take Apache and PHP as an example how to have a highly available web server.  As usual, I use FreeBSD for the purpose.

Why Apache and PHP!?

Quite some people would argue Apache and PHP are outdated.  I am not going to make a comment on this.  I hope you will appreciate the shortness of this article because of this choice.  The concepts you acquire from this example can be applied with your favourite platform.

Active / Active Web Servers

The example today is active / active pair.  The two servers can serve webpages together without error.  In fact, one can have unlimited amount of hosts working together.  In contrast, in the previous examples of NFS, only one server is active and another server is passively receiving updates and standby to take over.

To make this happen, one needs to ensure intermediate state of executions of one servers are saved so they can be taken up by the other servers.  In this example, we will move these state to the highly available storage.  With role separation, those intermedia data will not be lost no matter what happens to the web servers.  Another benefit is that web servers can be easily added (handle more load), replaced (for system updates), or reduced (for economy) without affecting any user sessions.

Session Data

When running a cluster of web hosting servers, one needs to take care of the session data.  HTTP servers are stateless by itself.  Cookies (notes to Europeans) are saved on the client side so servers need not to worry about.  Sessions storage, provided by most web programming environments, expect the storage on the server is consistent and persistent for each user session.

Session data is useful storing information that should not be understood or modify the the web users.  This could be some intermediate game states, some login privileges, etc.  It is therefore important to make sure such data is not lost when a user encounters another web server in the cluster.

Packages

At the time of writing, Apache 2.4 and PHP 7.1 are the newest.

In order to run the PHP in the most lazy way, install “mod_php” packages.  In addition, install some common PHP modules.  There is a moderate collection called “php71-extensions”.  I would also install “php71-mysqli” and “php71-gd”.  They are for database connections and image processing respectively.

# pkg install mod_php71 php71-mysqli php71-gd

What about Apache?  It is installed automatically as an dependency when you request installing the “mod_php”.

Shared Directories

To build on top of the previous example, I am mounting the NFS prepared previously to /mnt/nfs.  On the shell, overtime you want to mount:

# mkdir /mnt/nfs
# mount 10.65.10.13:/nfs /mnt/nfs
#

Alternatively, in the “/etc/fstab”:

# mkdir /mnt/nfs
# cat >> /etc/fstab << EOF
10.65.10.13:/nfs.   /mnt/nfs    nfs    rw    0    0
EOF
# mount /mnt/nfs
#

Configuring the Apache

Modify Apache so that the web page and script locations are in the NFS share.  The file to be modified is “/usr/local/etc/apache24/httpd.conf”.  There are originally two directories for normal web pages and CGI pages in “/usr/local/www/apache24”.  We are moving it to “/mnt/nfs”.

# sed -ibak 's|/usr/local/www/apache24|/mnt/nfs|' \
  /usr/local/etc/apache24/httpd.conf
# cp -a /usr/local/www/apache24/cgi-bin /mnt/nfs/
# cp -a /usr/local/www/apache24/data /mnt/nfs/
# cat >> /etc/rc.conf << EOF
apache24_enable="YES"
EOF
#

Also, update the “/usr/local/etc/apache24/httpd.conf” so that it decodes PHP files.  Modify the following parts manually:

<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>

<FilesMatch "\.php$">
    SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch "\.phps$">
    SetHandler application/x-httpd-php-source
</FilesMatch>

Configuring the PHP

PHP requires only copying the default configuration, and then modifying the session path.  One can also modify the upload path similarly, but I do not think it is necessary.

# cd /usr/local/etc/
# cp php.ini-production php.ini
# vi php.ini

The line to be added is just as follows:

; http://php.net/session.save-path
;session.save_path = "/tmp"
session.save_path = "/mnt/nfs/tmp"

Example Code

To make effect, reboot Apache24.  Here is a simple code that prints a counter every time it is loaded.  Write a file “/mnt/nfs/data/counter.php”

<?php
session_start();
if (isset($_SESSION['counter']))
  $_SESSION['counter'] += 1;
else
  $_SESSION['counter'] = 1;
print($_SESSION['counter']);
session_write_close();
?>

Testing

In order to test the service, we first turn on the web servers.  Then, we alternatively point a domain name to either one of the IP addresses.  The counter code above is repetitively executed as the map changes.  If successful, the counter continues increasing.

The laziest way to change the domain name mapping is of course editing the host table where the web browser is run.  For BSD and BSD-like systems, edit “/etc/hosts”.  For Windows, edit “C:\Windows\System32\Drivers\etc\hosts”.

Upcoming Steps

Once there are multiple web servers ready, one can consider having a load balancer or a content distribution network.  A load balancer service can be found in some value-added cloud service providers.  (Or else, you can set up one yourself.)  A content distribution network can be found anywhere around the globe.  Put aside their fundamental difference, they both accept multiple web server addresses and route network traffic accordingly.

Troubleshoot

Getting a blank page of PHP:  If you follow my instructions, you have configured PHP in a production mode.  Errors messages are not displayed but saved to the Apache log, which is “/var/log/httpd-error.log” in our context.  Read it and you will get an idea.

Function “session_start” undefined:  Make sure you have installed the package “php71-session” and restarted Apache.  It should be installed as an dependency when you install the package “php71-extensions”.

Highly Available MySQL Server

Standard

Previously, I have discussed how to setup a highly available block device and also a highly available file system.  In this article, I further demonstrate how to setup a highly MySQL database service.

Installing the Packages

As usual, one will need to install the package on two hosts and it can be easily done by:

# pkg install mysql57-server

I know there are alternatives.  Forgive my laziness.

Running for the First Time

We are starting MySQL once so that it generates the file structures.  Try to login, and then Ctrl-D to exit.

# service mysql-server onestart
# cat /root/.mysql_secret
# mysql -u root -p
Password: **********
root@localhost [(none)]> ^D
# service mysql-server onestop

It is then discovered (through educated guess) some directories are created in the “/var/db” directory, namely “mysql”, “mysql_tmpdir”, and “mysql_secure”.  Suppose you already have the “/db” mounted (as in the previous article), move them there and make the replacement symbolic links.

# mv /var/db/mysql /db
# mv /var/db/mysql_tmpdir /db
# mv /var/db/mysql_secure /db
# ln -s /db/mysql /var/db/
# ln -s /db/mysql_tmpdir /var/db/
# ln -s /db/mysql_secure /var/db/
# ls -ld /var/db/mysql*
lrwxr-xr-x  1 root  wheel   9 Apr 19 20:37 /var/db/mysql -> /db/mysql
lrwxr-xr-x  1 root  wheel  16 Apr 19 20:37 /var/db/mysql_secure -> /db/mysql_secure
lrwxr-xr-x  1 root  wheel  16 Apr 19 20:37 /var/db/mysql_tmpdir -> /db/mysql_tmpdir

Some would question why not change the configuration for the new paths.  I find it mostly a matter of taste.  If you want to make lives easier for those who have recited the default paths, do make the symbolic links.

Configurations

You will want to modify the configuration file “/usr/local/etc/mysql/my.cnf”.  For your reference, there is a sample file “my.cnf.sample”.  At minimum, you will need to modify the bind address (default 127.0.0.1) so that the service is available not just locally, but to the other computers in the same intranet.

The Script

The script for starting and stopping the MySQL server is simpler than the NFS one and are as follows.  Like last time, automatic switching is skipped due to my conflict of interest.  You will need a mechanism to call “start” and “stop” properly.

#!/bin/sh -x

start() {
 ifconfig vtnet1 add 10.65.10.14/24
 hastctl role primary db_block
 while [ ! -e /dev/hast/db_block ]
 do
 sleep 1
 done
 fsck -t ufs /dev/hast/db_block
 mount /dev/hast/db_block /db
 service mysql-server onestart
}

stop() {
 service mysql-server onestop
 umount /db
 hastctl role secondary db_block
 ifconfig vtnet1 delete 10.65.10.14
}

status() {
 ifconfig vtnet1 | grep 10.65.10.14 && \
 service mysql-server onestatus && \
 ls /dev/hast/db_block
}

residue() {
 ifconfig vtnet1 | grep 10.65.10.14 || \
 service mysql-server onestatus || \
 mount | grep /db || \
 ls /dev/hast/db_block
}

clean() {
 residue
 if [ $? -ne 0 ]
 then
 exit 0
 fi
 exit 1
}

if [ "$1" == "start" ]
then
  start
elif [ "$1" == "stop" ]
then
  stop
elif [ "$1" == "status" ]
then
  status
elif [ "$1" == "clean" ]
then
  clean
fi

Troubleshoot

If there are any issues MySQL fails to start, you can verify its absence with the command “service mysql-server onestatus”.  There are also log files located in the MySQL data directory; in our context, it is “/db/mysql/<hostname>.err”.  Please note the end of the log is most likely a graceful shutdown.  You will need to scroll upwards for the actual reason why the startup failed.

Highly Available Network File System

Standard

In the previous article, we discussed the way to create a highly available block device by replication.  We continue and attempt making a network file system (NFS) on top of it.  We first discuss the procedures to start and stop the service.  Then we have the script…  Some parts are deliberately missing due to my conflict of interest.

NFS Configuration

Since it is not our goal here, we only do minimal NFS configuration in this example.  In short, the export(5) file “/etc/exports” is being modified like as follows.  This implies the directory “/nfs” is shared with the given two IP subnets.

/nfs -network=10.65.10.0/24
/nfs -network=127.0.0.0/24

Unlike previous setting, we do not use the “/etc/rc.conf” file to start the service.  This is because we like to control when a service is started, instead of blindly just after boot.  In FreeBSD, services can be started with the “onestart” command.

Firewall Configuration

Configuring NFS for a tight firewall is tricky, because it uses random ports.  For convenience, a simple IP address-based whitelist can be implemented.  In this example, we have the server IP 10.65.10.13 (see later), and the client IP 10.65.10.21.  If you simply do not have a firewall, skip this part.  On the server side, the PF can be configured with:

pass in quick on vtnet1 from 10.65.10.21 to 10.65.10.13 keep state

On the client side, the PF can be configured with:

pass in quick on vtnet1 from 10.65.10.13 to 10.65.10.21 keep state

Starting the Service

When we start the service, we want the following to happen:

  1. Acquire the IP address, say 10.65.10.13, regardless which machine it is running.
  2. Activate the HAST subsystem so to become the primary role.
  3. Wait for the HAST device to be available.  If the device is in secondary role, the device file in “/dev/hast” will not appear so we can go to sleep a while.
  4. Run the file system check just in case the file system was corrupted in the last unmount.
  5. Mount the file system for use (in this example, “/nfs”)
  6. Start the NFS-related services in order: the remote procedural call binding daemon, the mount daemon, the network file system daemon, the statistic daemon, and the lock daemon.
  7. Once the step 5 completes, the service is available to the given clients as instructed to the NFS and allowed by the firewall.

For the inpatient, one can jump to the second last section for the actual source code.

Stopping the Service

Stopping the service is the reverse of starting, except some steps can be less serious.

  1. Stop the NFS-related services in order: the lock daemon, the statistic daemon, the network file system daemon, the mount daemon, and finally the remote procedural call binding daemon.
  2. Unmount the file system.
  3. Make the HAST device in secondary role.
  4. Release the iP address so neighbours can reuse.

Also, one can jump to the second last section for the actual source code.

Service Check Script

There are two types of checking.  The first one ensures all the components (like the IP address, mount point service, etc) are present and valid.  The procedure returns success (zero) only when the components are all turned on.  Whenever a component is missing, it will be reported as a failure (non-zero return code).

The second one ensures all the components are simply turned off, so that the service can be started on elsewhere.  The procedure returns success (zero) only when all the components are turned off.  Whenever a component is present, it will be reported as a failure (non-zero return code).

What is Missing

Once we master how to start and stop the service on one node, we need the mechanism to automatically start and stop the service as appropriate.  In particular, it is utmost important not to run the service concurrently on two hosts, as this may damage the file system and confuse the TCP/IP network.  This part should be done out of the routine script.

The Script

Finally, the script is as follows…

#!/bin/sh -x

start() {
  ifconfig vtnet1 add 10.65.10.13/24
  hastctl role primary nfs_block
  while [ ! -e /dev/hast/nfs_block ]
  do
    sleep 1
  done
  fsck -t ufs /dev/hast/nfs_block
  mount /dev/hast/nfs_block /nfs
  service rpcbind onestart
  service mountd onestart
  service nfsd onestart
  service statd onestart
  service lockd onestart
}

stop() {
  service lockd onestop
  service statd onestop
  service nfsd onestop
  service mountd onestop
  service rpcbind onestop
  umount /nfs
  hastctl role secondary nfs_block
  ifconfig vtnet1 delete 10.65.10.13
}

status() {
  ifconfig vtnet1 | grep 10.65.10.13 && \
  service rpcbind onestatus && \
  showmount -e | grep /nfs && \
  mount | grep /nfs && \
  ls /dev/hast/nfs_block
}

residue() {
  ifconfig vtnet1 | grep 10.65.10.13 || \
  (service rpcbind onestatus && showmount -e | grep /nfs) || \
  mount | grep /nfs || \
  ls /dev/hast/nfs_block
}

clean() {
  residue
  if [ $? -ne 0 ]
  then
    exit 0
  fi
  exit 1
}

if [ "$1" == "start" ]
then
  start
elif [ "$1" == "stop" ]
then
  stop
elif [ "$1" == "status" ]
then
  status
elif [ "$1" == "clean" ]
then
  clean
fi

Testing

To test, fine the designated computer and mount the file system.  Assume the file system has been running on the host “store1”, make a manual failover to see…  The file client does not need to explicitly remount the file system; it cab be remounted automatically.

client# mount 10.65.10.13:/nfs /mnt
client# ls /mnt
.snap
client# touch /mnt/helloworld
store1# ./nfs_service.sh stop
store2# ./nfs_service.sh start
client# ls /mnt
.snap helloworld

Highly Available Storage Target on FreeBSD

Standard

To achieve highly available service, it is vital to have the latest data available for restarting the service elsewhere.  In enterprise environments, multipath SAS drives allows each drive to be accessible from multiple hosts.  What about the rest of us, living in the cloud of / or inexpensive SATA drives?  Highly Available Storage Target (HAST) is the answer.  It relatively a new feature in FreeBSD 8.1.  It is useful to keep two copies of drive content on two loosely coupled computers.

In this article, I demonstrate how HAST can be setup, without bothering the actual failover logic.  I assume the two virtual machines are prepared from scratch with some unallocated space.  Alternatively, you can prepare a virtual machine with multiple drives (which is not quite feasible in my setting).

Preparing the Partitions

First, examine the drives.  Here we have 18 (no, indeed 17.9 something) gigabytes of space available.

# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  37748696       - free - (18G)

Then, we can add two more partitions.  Since we have two hosts, we set two partitions so that each host can run the service on one partition.  This is so-called active-active setup.  I dedicate one for NFS and one for database.  Once you finish working on one virtual machine, do not forget performing the same on another machine.

# gpart add -t freebsd-ufs -l nfs_block -a 1M -s 8960M /dev/vtbd0
vtbd0p4 added.
# gpart add -t freebsd-ufs -l db_block -a 1M -s 8960M /dev/vtbd0
vtbd0p5 added.

This is the result.  Two block devices are created and made available inside /dev/gpt with their appropriate labels.

# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  18350080     4 freebsd-ufs (8.8G)
  33030144  18350080     5 freebsd-ufs (8.8G)
  51380224   1048536       - free -  (512M)
# ls /dev/gpt
db_block nfs_block

HAST Daemon Setup

Here is an sample of defining the HAST configuration, hast.conf(5).  In short, there are two hosts and two resource items.  The host “store1” has its remote partner “store2” and vice versa.  Since we use the short host names “store1” and “store2”, do not forget to update the host(5) file to make them resolvable.  Remember to repeat these another machine.  Thankfully, the HAST configuration need not to be customised for each member host.

# sysrc hastd_enable="YES"
# cat > /etc/hast.conf << EOF
resource nfs_block {
  on store1 {
    local /dev/gpt/nfs_block
    remote store2
  }
  on store2 {
    local /dev/gpt/nfs_block
    remote store1
  }
}
resource db_block {
  on store1 {
    local /dev/gpt/db_block
    remote store2
  }
  on store2 {
    local /dev/gpt/db_block
    remote store1
  }
}
EOF
# service hastd start

Firewall Rules

If you are having a firewall, remember to open the port number 8457 opened.  For example, in PF, add these three lines to the two hosts.  Remember the replace the IP addresses as appropriate.

geompeers="{10.65.10.11,10.65.10.12}"
geomports="{8457}"
pass in quick inet proto tcp from $geompeers to any port $geomports keep state

HAST Daemon Status

Once the HAST daemon is started, the status of the blocks can be checked.  Since we have defined two resource items, there are two HAST device status reported.  For example, in the host “store1”, it says there are two components for each resource item: one block device, and one remote host.  At first, the resource items are in “initialisation” state:

store1# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store2
db_block  -        init      /dev/gpt/db_block store2

To turn on the device for operation, use the “role” subcommand on “store1”

store1# hastctl role primary db_block
store1# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store2
db_block  degraded primary   /dev/gpt/db_block store2

Similarly, use the “role” command on “store2”, but this time we set it secondary:

store2# hastctl role secondary db_block
store2# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store1
db_block  -        secondary /dev/gpt/db_block store1

When the synchronisation completes, the status is marked complete:

store2# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store1
db_block  complete secondary /dev/gpt/db_block store1

Formatting the Block Devices

We can format the block device like usual, just that we should format the device under the device directory “/dev/hast” instead of “/dev/gpt”.  Since currently the “db_block” is active on the host “store1”, it has to be executed over there.

store1# newfs -J /dev/hast/db_block
/dev/hast/db_block: 8960.0MB (18350064 sectors) block size 32768, fragment size 4096
using 15 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
super-block backups (for fsck_ffs -b #) at:
192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632, 8975872, 10258112, 11540352, 12822592, 14104832, 15387072, 16669312, 17951552

One thing to note is, the raw device was 18350080 sectors.  When HAST takes it for service, there is only 18350064 blocks left for payload.  Next, we can mount the file system.  We do not use the fstab(5) like before, because they do not need to execute every time boot.

store1# mkdir /db
store1# mount /dev/hast/db_block /db

Switching Over

In order to switch over, the procedure is as follows.

store1# umount /db
store1# hastctl role secondary db_block
store2# hastctl role primary db_block
store2# mkdir /db
store2# mount /dev/hast/db_block /db

What if, in an error, the backend device in “/dev/gpt” is being used for mounting?  It will say the following.  The chance to go wrong is really not heavy.

store2# mount /dev/gpt/db_block /db
/dev/gpt/db_block: Operation not permitted

For automatic switching, it will be discussed in a separated article.

Troubleshooting

Nothing can go really wrong without a serious application on top.  Nevertheless, the following message troubled me for a few hours.

We act as primary for the resource and not as secondary as secondary“: check the HAST configuration.  Likely a host is configured to have itself as the remote partner.

To be Continued

In the upcoming articles, I will cover how to make use of this highly-available block storage for shared file system, database, and web servers.