Let's Balance the load Part 2 - L3/L4

At L3/L4 level we use virtual IP address to expose our services to the clients and these virtual IPs forwards the traffic to actual servers based on various load balancing algorithms.

Creating a basic L3/L4 Load Balancer

There are multiple ways to create a L3/L4 Load balancer for simplicity sake we will use iptables which is available on most of the unix systems.

Let’s assume we have 4 servers 192.168.1.1, 192.168.1.2, 192.168.1.3 and 192.168.1.4. We have a load balancer machine with IP 10.10.10.1. Now, we want to distribute the incoming traffic on port 80 across these servers.

This can be done in 4 steps per upstream server:

1. Tell iptables that we want to apply Network Address Translation (NAT) to incoming traffic on port 80 as the packets come to our load balancer IP.

   iptables -A PREROUTING -t nat -p tcp -d 10.10.10.1 --dport 80 -m statistic --mode nth --every 4 --packet 0 -j DNAT --to-destination 192.168.1.1:80
   

   • -p tcp tells iptables to apply this to tpc traffic only.
   • -m statistic tells iptables command to use statistic module.
   • --mode nth tells it to use nth mode which instructs iptables to apply this rule to every nth packet.
   • --every 4 means that every fourth packet will be sent to 192.168.1.1.
   • --packet 0 makes sure that rule should be applied to first packet of the stream and all the packets in that stream should be sent to the same destination.

2. Tell iptables to perform reverse NAT so that packets coming from these servers can reach to the source IP correctly.

   iptables -A POSTROUTING -t nat -p tcp -d 192.168.1.1 --dport 80 -j SNAT --to-source 10.10.10.1
   

3. Tell iptables to ACCEPT the packets coming to port 80 from Clients.

   iptables -t filter -A FORWARD -p tcp -d 192.168.1.1 --dport 80 -j ACCEPT
   

4. Tell iptables to ACCEPT the packets coming to port 80 from Servers.

   iptables -t filter -A FORWARD -p tcp -s 192.168.1.1 --sport 80 -j ACCEPT
   

We will repeat above steps for all the upstream servers except for the last one. We want to send all the remaining traffic to the last server. To do this we will execute the following command in step 1. Rest of the steps remain same.

iptables -A PREROUTING -t nat -p tcp -d 10.10.10.1 --dport 80 -j DNAT --to-destination 192.168.1.4:80

All these steps needs to be executed on Load Balancer Server.

Put it together

Our final iptables rules will look like this:

# On 10.10.10.1

# Destination 192.168.1.1
iptables -A PREROUTING -t nat -p tcp -d 10.10.10.1 --dport 80 -m statistic --mode nth --every 4 --packet 0 -j DNAT --to-destination 192.168.1.1:80
iptables -A POSTROUTING -t nat -p tcp -d 192.168.1.1 --dport 80 -j SNAT --to-source 10.10.10.1
iptables -t filter -A FORWARD -p tcp -d 192.168.1.1 --dport 80 -j ACCEPT
iptables -t filter -A FORWARD -p tcp -s 192.168.1.1 --sport 80 -j ACCEPT

# Destination 192.168.1.2
iptables -A PREROUTING -t nat -p tcp -d 10.10.10.1 --dport 80 -m statistic --mode nth --every 3 --packet 0 -j DNAT --to-destination 192.168.1.2:80
iptables -A POSTROUTING -t nat -p tcp -d 192.168.1.2 --dport 80 -j SNAT --to-source 10.10.10.1
iptables -t filter -A FORWARD -p tcp -d 192.168.1.2 --dport 80 -j ACCEPT
iptables -t filter -A FORWARD -p tcp -s 192.168.1.2 --sport 80 -j ACCEPT

# Destination 192.168.1.3
iptables -A PREROUTING -t nat -p tcp -d 10.10.10.1 --dport 80 -m statistic --mode nth --every 2 --packet 0 -j DNAT --to-destination 192.168.1.3:80
iptables -A POSTROUTING -t nat -p tcp -d 192.168.1.3 --dport 80 -j SNAT --to-source 10.10.10.1
iptables -t filter -A FORWARD -p tcp -d 192.168.1.3 --dport 80 -j ACCEPT
iptables -t filter -A FORWARD -p tcp -s 192.168.1.3 --sport 80 -j ACCEPT

# Destination 192.168.1.4
iptables -A PREROUTING -t nat -p tcp -d 10.10.10.1 --dport 80 -j DNAT --to-destination 192.168.1.4:80
iptables -A POSTROUTING -t nat -p tcp -d 192.168.1.4 --dport 80 -j SNAT --to-source 10.10.10.1
iptables -t filter -A FORWARD -p tcp -d 192.168.1.4 --dport 80 -j ACCEPT
iptables -t filter -A FORWARD -p tcp -s 192.168.1.4 --sport 80 -j ACCEPT

I’ve written a small python script which automates all the above steps and makes it easier to use. You can find the script here.

Testing

To test the load balancer you can host Nginx on those servers with custom html file.

Let’s create a simple HTML file for each server. For example:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to Server 1!</h1>
</body>
</html>

Change the server number for every server to know which server html file is coming from.

Start Nginx server on each server:

# On 192.168.1.1
docker pull nginx
docker run -it --rm -d -p 80:80 --mount type=bind,source="$(pwd)"/server1-index.html,target=/usr/share/nginx/html/index.html,readonly --name server1 nginx

# On 192.168.1.2
docker pull nginx
docker run -it --rm -d -p 80:80 --mount type=bind,source="$(pwd)"/server2-index.html,target=/usr/share/nginx/html/index.html,readonly --name server2 nginx

# On 192.168.1.3
docker pull nginx
docker run -it --rm -d -p 80:80 --mount type=bind,source="$(pwd)"/server3-index.html,target=/usr/share/nginx/html/index.html,readonly --name server3 nginx

# On 192.168.1.4
docker pull nginx
docker run -it --rm -d -p 80:80 --mount type=bind,source="$(pwd)"/server4-index.html,target=/usr/share/nginx/html/index.html,readonly --name server4 nginx

Now try to load the website using ip address of the load balancer machine.

curl http://10.10.10.1:80/

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to Server 1!</h1>
</body>
</html>

Running the curl command again results in response from another server:

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to Server 2!</h1>
</body>
</html>

Conclusion

• We implemented a basic L3/L4 load balancer using iptables.
• The load balancer distributes incoming traffic based on the round-robin algorithm.
• Nginx server sees that all the traffic is coming from the same IP address i.e. the IP of the load balancer (10.10.10.1).