Load balancing and replicating multiple servers has a great array of benefits, though orchestrating and keeping them in sync can be very tricky. Here, we will walk through some of the load balancing options available, as well as setting up a very basic one-way replication sync between two or more servers behind a load balancer.

What is server replication?

Load balancing is a way to increase the processing power and redundancy of your web application by spreading the traffic among multiple different servers. Traffic is orchestrated by a load balancer and the web nodes are orchestrated by another data replication mechanism. That is to say, the load balancer itself has nothing to do with data replication; it only routes traffic to the web nodes. Something else is necessary to keep the web nodes’ data and configuration in sync, which is server replication.

There are a variety of methods for syncing files between web servers. These fall into four overlapping categories, which are synchronous and asynchronous, as well as one-way and two-way. Most synchronous sync types are two way, while asynchronous sync types could be one-way or two-way.

Synchronous sync types instantly share files between servers, via a shared storage node (such as a SAN or object store), and/or by coordinating a shared local or remote file system (OCFS2). These methods are complicated to configure, since beyond mounting the file systems at the same time, all servers must also communicate about when they are ready to write files and write to all locations at the same time. Liquid Web offers Managed Replication in our Enterprise Hosting offerings, removing the planning and maintenance burden from your shoulders so you can focus on your application.

Asynchronous sync types are simpler to set up but do not share files instantaneously. After files are completely written to one location, they are pushed out to another location by a service running on that server (lsyncd) or by a regularly timed sync cron. These are generally set up in one direction, so that a master server replicates out to slave servers.

What are the advantages and drawbacks of server replication?

As I mentioned, load balancing multiple servers serving the same set of data will increase the processing power behind your website, as well as introduce some redundancy. If one of your replicated nodes fails, the other (or others) can continue to serve traffic while the failed node is repaired or replaced. Load balanced systems can also be scaled easily; nodes can be added when more traffic is expected, and taken down if they are no longer needed.
But, with more servers comes more configuration. Keeping all of the server nodes in sync with each other requires additional applications to be set up and running, as well as adding additional hardware for load balancing. It is also a good idea to dedicate the web nodes to Apache only, so database information should be offloaded to a separate server as well. Further, though the software may be freely available, multiple servers and appliances cost more money to run than a single server, so load balanced clusters are inherently more expensive.

What are the requirements for setting up replication?

To set up server replication, you need two servers and one load balancer at bare minimum. But it is recommended to have a separate database server or cluster as well, to further increase redundancy.

You should also plan for the type of replication you want to use. Some replication types require additional hardware or configuration beyond what is covered here. If you are interested in those types of replication, chat with our architects about our Managed Replication products.

For the purposes of this article, we will use Liquid Web’s Cloud Load Balancer, along with two core-managed VPS web servers, and one core-managed VPS database server, connected to each other via the Cloud Private Network with private IPs. We will call the web nodes web01 and web02, and the database node db01. The web01 node is set up with password-less SSH keys into web02. All servers have one public IP.

We will also assume that Apache is set up on each web server for name-based virtual hosting and is installed at /etc/httpd/. PHP is also installed on both machines, and their configuration files are static and matched. Finally, our database server has mariadb installed and running, and the firewall is open for external mysql connections.

Step 1: Set Up Apache

In order to sync the Apache VirtualHost files, we will set up a single folder with single configuration files for each domain. In our main Apache config file at /etc/httpd/conf/httpd.conf, we will add the following line at the very end:

IncludeOptional vhosts/*.conf

This will allow Apache to load configuration files from the /etc/httpd/vhosts/ folder, if there are any, which we will make next. From the command line, run:

mkdir /etc/httpd/vhosts

Perform these steps on both web01 and web02.

In this folder, we can set up our individual domains’ configuration files. Make sure they end in .conf so that Apache will load them. In our example, we will make two files: domain.com.conf and domain.net.conf. These both will be set up with valid VirtualHost blocks, which we won’t get into the details of here. For this exercise, the docroots of the domains are at /var/www/domain.com/ and /var/www/domain.net/. We only need to make these configuration files on web01 for now, since we will sync them later.

Step 2: Set Up Databases

If your application requires it, set up databases on your dedicated database server, and add grants so that all of the web nodes can connect. For instance, after configuring the database user with a good strong password, you might run the following grant statement if your web nodes had private IPs of 192.168.0.11 and 192.168.0.12:

mysql -e “grant all privileges on your_db.* to your_user@’192.168.0.11’; grant all privileges on your_db.* to your_user@’192.168.0.12’”

To introduce elasticity, and if you are sure that all of your private IPs will have the same prefix, you might consider running this instead:

mysql -e “grant all privileges on your_db.* to your_user@’192.168.0.%’”

This has allowed all IPs that start with 192.168.0 to access this database, if they have valid credentials. Now, if you add another web node, it will also be able to access the database without additional grants being made.

We can now connect the web node to the database using its configuration file. On WordPress, for instance, this file is wp-config.php. Enter the appropriate connection credentials, using the private IP of the database server as the DB_HOST. We use the private IP so that we don’t waste public bandwidth on MySQL communication. Setting up the host connection to a hostname or IP like this will help ensure that the wp-config.php file will work on all your web servers the same way.

Step 3: Install and Configure LSyncD

For our asynchronous one-way replication, we will use Live Sync Daemon (lsyncd). This is a freely available daemon which can watch a folder for activity, and then replicate that activity with rsync in another local or remote location. We need to add the EPEL repository to install it via yum:

yum -y install epel-release
yum -y install lsyncd

Now that lsyncd is installed, we can configure it for each folder we want to sync to other nodes. In this case, we will be syncing the vhost directory we made earlier, as well as the docroots for each domain. In the /etc/lsyncd.conf file, delete the example sync command, and set up the following block of data:

sync {
default.rsyncssh,
source = "/var/www/domain.com",
host = "192.168.0.12",
targetdir = "/var/www/domain.com",
rsync = {
binary = "/usr/bin/rsync",
archive = true,
hard_links = true,
update = true
}
}

Notice the required commas after all but the last configuration lines. This is because the sync command can also be set up on one line. Make sure you don’t add a comma after the final element of any curly brace array.

Based off of this block, we can set up similar blocks for /var/www/domain.net and any other running sites. Just add them all one after the other in the /etc/lsyncd.conf file.

Lastly, enable and start lsyncd:

systemctl enable lsyncd
systemctl start lsyncd

You can test that it’s working by checking out the contents of the directory on web02. Everything should be there! Make an update to a file, and check the target to see how long it takes to arrive. It should take just 5-10 seconds for lsyncd to pick up the change and copy the file.

Step 4: Set Up Apache Configuration Replication

All of the above is sufficient for servers that do not regularly change the number of domains they host. But, if you configure new domains or update your Apache configuration frequently, you need to set up a script to help you sync this and restart Apache on the other servers. We can call this from lsyncd as the rsync execution binary and have it run post-sync tasks. Create a file called /root/vhostsync.sh that looks like this:

#!/bin/bash
/usr/bin/rsync "$@"
[ $? -eq 0 ] && ssh 192.168.0.12 “systemctl reload httpd”

Exit your editor, then add execute permissions to the file:

chmod 700 /root/vhostsync.sh

This wrapper will perform the rsync task for you with the arguments passed by lsyncd, and if that is successful, connect to web02 by IP and reload the apache configuration. Now, head back into /etc/lsyncd.conf and add this sync block:

sync {
default.rsyncssh,
source = "/etc/httpd/vhosts",
host = "192.168.0.12",
targetdir = "/etc/httpd/",
rsync = {
binary = "/root/vhostsync.sh"
}
}
Since we declare our script as the rsync binary, this script will execute instead of running rsync by itself. Reload lsyncd to add this sync to the running config:

systemctl restart lsyncd

Once it reloads, lsyncd should copy the folder over to web02 and reload Apache. And, since we already set up the folder to be included, web02 should now also be serving the two virtualhost blocks we made for domain.com and domain.net.

Step 5: Route Traffic

That should just about do it! The final step is to test the configuration by connecting directly to web01 and web02 using hosts file modification, and make sure the content being served from both machines is the same. Also ensure that both servers can properly reach their database on db01.

Now, create the Cloud Load Balancer to route traffic to the IPs for web01 and web02. Use your hosts file again to connect to the load balancer’s VIP to ensure you can reach the nodes.
After testing, public DNS for the hosted domains can be changed to the VIP of the load balancer, allowing traffic to be routed into your cluster. Done!

Caveats

As you may have noticed, this only sets up replication from web01 to web02, not vice versa. Therefore, if you upload content or make changes to your website, you should always do so from web01. It is possible in some Load Balancer solutions to route traffic for your website’s administrative panel, such as /wp-admin, to only one of your nodes, i.e. web01. This will allow content creators to properly upload data only to the master web node for correct replication. This is not an option on the Cloud Load Balancer, though it is available in our Shared and Dedicated Load Balancer offerings.

If you decide to add a new domain to the servers, you will need to do all the following:

• • Create the document root on web01
  • Create any databases on db01 and set up SQL grants
  • Add a new virtualhost file to the /etc/httpd/vhosts/ folder
  • Add a new sync statement to /etc/lsyncd.conf for the new docroot
  • Restart lsyncd and apache on web01

Additionally, when adding or removing a server, new blocks will need to be set up manually inside of /etc/lsyncd.conf to sync to this new host too, as well as a new vhostsync wrapper for restarting Apache on those new nodes after configuration sync.

Finally, though this does provide some redundancy, your web01 server is still the master node, since it is running lsyncd. If web01 goes down, your website will still work, but you ought not add to or modify your sites until it can be repaired or replaced.

Do I Still Need Backups?

Absolutely!

Server replication is not a stand-in for server backups. If a hard disk crashes or your RAM goes bad, the other web node will still be able to stand up traffic. However, if your website is compromised, or you accidentally delete files, these changes will be replicated over to the other web node, removing your “backups”. Make sure you have local and full-server backups for disaster recovery.

If you are interested in setting up your own load balanced server replication, or would like more information about our managed products to help you with synchronous or asynchronous replication, chat with a hosting advisor today!