We get familiar with load balancing and learn how to use Nginx as a load balancer.
Load balancing means distributing incoming traffic (for example, users' requests) across multiple servers so that no single server is put under too much load and the system's performance is maintained.
Distributing traffic across multiple servers increases our services' availability as well as their scalability.
High Availability: if one of the servers goes down, requests are routed to the other servers.
Scalability: by adding new servers, you can easily handle a larger volume of requests.
There are various tools for load balancing (distributing traffic), and one of the most common is nginx. Using nginx, you can load balance HTTP, TCP, and even UDP traffic.
Load Balancing HTTP Requests
In the previous parts, we said that the /etc/nginx/nginx.conf file is where nginx loads the rest of the configuration from. Let's take a look at this file:
We can see there's an http block. The http block handles configuration related to http requests. On the last line of this block, the include directive loads every file with a conf extension as configuration from /etc/nginx/conf.d/. This means any file with a conf extension we put at /etc/nginx/conf.d/ is used as configuration for http requests.
Using the knowledge we've gained so far, we can implement a load balancer with nginx to distribute requests across multiple servers. Suppose several active instances of an application are running, and we want to load balance incoming traffic between them. To do this, we can simulate three servers and use an nginx as the load balancer to manage and distribute traffic.
To simulate this, let's create three servers using nginx itself. To do this, you can create a file named mock-server.conf at /etc/nginx/conf.d/ and put the settings below inside it:
No comments have been submitted for this article yet!
# service 1
server {
listen 8080 default_server;
location / {
default_type text/html;
return 200 '
<!DOCTYPE html>
<html>
<head>
<link rel="icon" href="data:image/png;base64,iVBORw0KGgo=">
</head>
<body style="background-color: #ff6b6b; color: white; text-align: center;">
<h1>Hello from Server 8080</h1>
</body>
</html>
';
}
}
# service 2
server {
listen 8081 default_server;
location / {
default_type text/html;
return 200 '
<!DOCTYPE html>
<html>
<head>
<link rel="icon" href="data:image/png;base64,iVBORw0KGgo=">
</head>
<body style="background-color: #4facfe; color: white; text-align: center;">
<h1>Hello from Server 8081</h1>
</body>
</html>
';
}
}
# service 3
server {
listen 8082 default_server;
location / {
default_type text/html;
return 200 '
<!DOCTYPE html>
<html>
<head>
<link rel="icon" href="data:image/png;base64,iVBORw0KGgo=">
</head>
<body style="background-color: #56ab2f; color: white; text-align: center;">
<h1>Hello from Server 8082</h1>
</body>
</html>
';
}
}
Using the directives above, we created three nginx servers that respond to incoming requests on ports 8080, 8081, and 8082, respectively. Using a hash sign, or #, you can add comments. Comments are left for developers so that later, by reading them, they can better understand the configuration that was written.
Now let's create a load balancer and configure it to distribute requests among them (I'll name my configuration file load-balancer.conf):
upstream backend01 {
server localhost:8080 weight=1;
server 127.0.0.1:8081 weight=1;
server 127.0.0.1:8082 weight=1;
}
server {
listen 8015;
location / {
proxy_pass http://backend01;
}
}
In this example, using upstream, we define a group of servers named backend01. We specify the servers' addresses with the server keyword, and using weight, we determine the ratio in which the traffic load is distributed among them.
Inside the upstream block, when defining servers, you can use an IP or a URL to specify the desired address.
In the example above, the weight is equal for all the servers specified, so requests are distributed evenly among them. If the weight is increased, requests are distributed onto that server proportionally more as well.
Next, we've created a server block that receives all incoming requests on port 8015. If the request address matches the path defined in the location section, the request is proxied to the servers in the backend01 group. This forwarding is done using the proxy_pass directive.
To match all requests, we use the / sign at the start of the location block. In this case, if we open localhost:8015 or 127.0.0.1:8015 in a browser, each time we send a request, we connect to one of the three servers we set up earlier, and the response comes back from that same server.
In some cases, one of the servers may have higher processing power and be able to handle more requests. In that case, by increasing the weight value in the upstream block, you can allocate a larger share of the traffic to that server.
upstream backend02 {
server localhost:8080 weight=1;
server 127.0.0.1:8081 weight=1;
server localhost:8082 weight=3;
}
server {
listen 8016;
location / {
proxy_pass http://backend02;
}
}
Load Balancing TCP and UDP
For TCP and UDP load balancing, you can use the upstream and server blocks just like before, except these blocks need to sit inside another block called stream. To do this, you can add the directives below to the end of /etc/nginx/nginx.conf:
stream {
include /etc/nginx/stream_conf.d/*.conf;
}
This way, all the configuration we define at /etc/nginx/stream_conf.d/ will be used to handle TCP and UDP requests.
For example, for TCP load balancing, suppose we have two MySQL database servers and we want to distribute requests between them. To do this, we can define a file named mysql-db-load-balancer.conf at stream_conf.d, and then put the settings below inside it:
upstream mysql-nodes {
server mysql-node-01:3306 weight=1;
server mysql-node-02:3306 weight=1;
}
server {
listen 3305;
proxy_pass mysql-nodes;
}
In the example above, we didn't use location, because in TCP requests, the request path no longer has any meaning.
For load balancing in udp mode, we just need to use the word udp in listen so Nginx knows it should receive requests on the UDP protocol.
upstream my-nodes {
server 127.0.0.1:2200 weight=1;
server 127.0.0.1:3200 weight=1;
}
server {
listen 3200 udp;
proxy_pass my-nodes;
}
In TCP and UDP load balancing, we need to keep in mind that Nginx has the following behavior by default:
Once a TCP or UDP connection is opened, it keeps its connection to the server it was connected to after load balancing, until the client itself closes the connection.
Because TCP or UDP connections are stateful, once opened they stay persistent. This means that if the client application (or, for example, a MySQL client) does connection pooling or keeps a connection open, that same connection effectively stays connected to one backend, because Nginx doesn't do further load balancing while the connection is open.
Load Balancing Algorithms
The algorithms that nginx supports in its free version are as follows:
Round robin
Least connection
Generic hash
Random
IP hash
Load Balancing With the Round Robin Algorithm
With the Round Robin approach, requests are distributed among the available servers in turn, in a rotating fashion; this is done regardless of how much load is already on each server or how fast it responds to requests.
In nginx, round robin is used by default. So if you haven't chosen another approach, the load is distributed across your servers using round robin.
upstream backend02 {
server localhost:8080 weight=1;
server localhost:8082 weight=3;
}
server {
listen 8016;
location / {
proxy_pass http://backend02;
}
}
Load Balancing With the Least Connection Algorithm
With this approach, every new request is sent to the server that currently has the fewest active connections. That is, Nginx first checks the number of active requests on each server, and then sends the new request to the server with the least current load.
This approach is suitable when requests' response times vary, meaning some requests are long and some are short. Servers with higher processing power can usually process requests faster, so they end up with fewer active connections. In that case, by sending more requests to those servers, Nginx tries to balance the load evenly across all the servers and keep the number of active connections at a proportionate level.
upstream backend02 {
least_conn;
server localhost:8080 weight=1;
server localhost:8082 weight=3;
}
server {
listen 8016;
location / {
proxy_pass http://backend02;
}
}
Load Balancing With the Generic Hash Algorithm
Generic Hash Load Balancing (or hash-based load balancing) works based on a hash function. This approach is mostly useful when we want similar requests to always be routed to the same specific server.
In this approach, Nginx takes a specific value from each request (such as the client's IP, the URL, or any parameter you want), and the result of hashing that value determines which server the request is sent to.
upstream backend02 {
hash $remote_addr;
server localhost:8080 weight=1;
server localhost:8082 weight=3;
}
server {
listen 8016;
location / {
proxy_pass http://backend02;
}
}
When we use the Hash algorithm for load balancing, one important thing to note is that if a server is added to or removed from the pool for any reason, the hash structure gets completely thrown off and the routing of most requests changes. This means users who were previously routed to a specific server might now be moved to other servers. To make this happen less often, we can use the consistent parameter. This parameter makes the distribution of requests change as little as possible when servers are added or removed.
upstream backend02 {
hash $remote_addr consistent;
server localhost:8080 weight=1;
server localhost:8082 weight=3;
}
server {
listen 8016;
location / {
proxy_pass http://backend02;
}
}
Load Balancing With the Random Algorithm
In this approach, Nginx randomly picks a server from the list of servers in upstream and sends the request to it. This algorithm acts in a purely random way, without taking the number of active connections or response speed into account. In this approach, you can also specify a weight for the defined servers.
upstream backend02 {
random;
server localhost:8080 weight=1;
server localhost:8082 weight=3;
}
server {
listen 8016;
location / {
proxy_pass http://backend02;
}
}
In the random approach, you can add certain methods after random to specify how servers are randomized. The only method available in the free version of nginx is two. If we specify the request distribution as random two;, nginx first randomly picks two servers from the list of servers we specified in upstream, and then distributes traffic between these two servers using least_conn. This way, two servers are always chosen, and traffic is sent to whichever one has fewer active connections.
Load Balancing With the IP Hash Algorithm
When using the IP hash approach, nginx generates a hash value based on the client's IP address, and always routes requests from a client with that same IP to a fixed server.
This approach guarantees that clients are proxied to the same upstream server, as long as that server remains available.
This approach is typically useful when we need a client to always communicate with a specific server. For example, when there's server-side caching, or when sessions (usually used for authentication) are stored locally on each server.
We should also keep in mind that if most clients come from a specific IP range, incoming traffic may be distributed unevenly across the servers.
In Nginx, the IP Hash approach can only be used for HTTP. You can also specify a weight for the defined servers.
upstream backend02 {
ip_hash;
server localhost:8080 weight=1;
server localhost:8082 weight=3;
}
server {
listen 8016;
location / {
proxy_pass http://backend02;
}
}
Health Checks
In nginx, there are two approaches for checking server health: passive and active. The passive approach is available in the free version of nginx.
With the Active approach, the load balancer periodically sends requests directly to check the servers' status. That is, nginx actively checks the servers to see whether they're still available and healthy.
With the Passive approach, the load balancer doesn't directly check the servers' status; instead, it only notices a problem when an error occurs while processing real user requests. In other words, nginx only notices a server failure when it receives error responses from that server during the flow of real user requests.
The free version of Nginx only supports the Passive approach. To configure a passive health check, we can do the following:
upstream backend {
server backend1.example.com:1234 max_fails=3 fail_timeout=3s;
server backend2.example.com:1234 max_fails=3 fail_timeout=3s;
}
Using the settings above, the upstream server's health is checked passively, by monitoring the responses received to client requests. If requests sent to the upstream server (named backend here) run into errors or time out, nginx figures out that the defined upstream server isn't healthy, and it stops sending requests to it.
The max_fails and fail_timeout parameters relate to error detection in the passive health check of upstream servers. These two help Nginx figure out when a backend server has become temporarily unavailable.
max_fails=3 means that if, within the specified time window (which is fail_timeout), Nginx gets no response or an error (such as timeouts or a 5xx error) from this server three times in a row, it considers that server "failed," or temporarily down.
fail_timeout=3s is the time window during which Nginx counts these errors. It's also the amount of time that, after a failure is detected, Nginx takes that server out of rotation and tries to connect to healthy servers for subsequent requests. After this time passes, Nginx tries connecting to that server again.
Suppose Nginx tries to connect to backend1 three times within 3 seconds and gets an error all three times. Nginx concludes that backend1 is currently down and stops sending requests to that server. After 3 seconds, it sends a test request to it again. If that succeeds, it makes that server active again.
You can access the sample code for the examples in this part on GitHub.