split_clients can be used for Blue/Green or Canary deployments.

• Blue/Green: In this approach, two separate, identical environments (one called Blue and the other Green) are kept ready. The application's current version runs on the Blue environment, and the new version is deployed to the Green environment. Once the new version's correct behavior on Green is confirmed, all traffic is switched over from Blue to Green at once. This approach reduces downtime and the risks associated with releases, since you can quickly roll back to the previous environment if a problem occurs.

• Canary: In this approach, the version containing the changes is first rolled out to a small portion of users, so its performance and stability in the real environment can be carefully monitored and evaluated. If it's successful, a gradually larger percentage of traffic is routed to the new version, until it eventually replaces the old version entirely.

Limiting the Number of Connections (connection limit)

To limit the number of active connections from each client, you can use the limit_conn directive. This directive is used together with the limit_conn_zone and limit_conn_status directives.

The limit_conn_zone directive defines a shared memory zone for storing information about active connections. It also specifies which key connections are counted by; for example, you can use the user's IP or a session ID as the key.

Here, the $binary_remote_addr variable is used, which represents the client's IP address in binary form so it takes up less space. With the zone=limitbyaddr option, we set the memory zone's name, and we've set its size to 10 megabytes.

The limit_conn_status directive specifies which HTTP status code is returned to a client if their number of connections exceeds the allowed limit. In this example, code 429 (Too Many Requests) is used, indicating too many requests.

Finally, the limit is applied using the limit_conn directive. In this directive, we first enter the name of the memory zone defined for managing connections (limitbyaddr, here), and then specify the maximum number of connections allowed per client, which in this example is 40 simultaneous connections.

The limit_conn and limit_conn_status directives can be used inside http, stream, and location blocks, whereas the limit_conn_zone directive can only be used inside the http block.

By default, the value of limit_conn_status is 503 (Service Unavailable).

Limiting the Rate of Requests (rate limit)

To limit the rate of requests per client, you can use the limit_req directive. This directive is used together with limit_req_zone and limit_req_status.

In the example above, we allocated 10MB of memory to store rate limit information. The rate limit is also based on users' IPs, which are stored in memory in binary form. We also specified that a user can make 3 requests per second, and if that's exceeded, the remaining requests will get a 429 status response.

In the limit_req directive, the zone value must always be specified. There are other values, such as burst, delay, and nodelay, that can be used if needed.

• burst=12: this value specifies how many extra requests (beyond the allowed rate) a given IP is allowed to send back-to-back (without being rejected right away) before Nginx starts rejecting requests. Here, 12 means that each user, by their IP, can send up to 12 extra requests in a short window, which temporarily wait in a queue.

• delay=9: this means the first 9 extra requests (from that same burst) are processed with a delay (that is, they're let through slowly, at the rate of 3 requests per second). But if the number of queued requests goes beyond 9 (that is, we reach requests 10, 11, and 12), those requests are accepted without delay (though they're still within the burst limit). If more than 12 extra requests come in (that is, the burst is used up), requests will be rejected with a 429 Too Many Requests error.

If we use nodelay, requests are accepted quickly, one after another, until the burst capacity is full. Only once the number of requests exceeds burst will requests be rejected with a 429 error.

To better understand burst together with delay and nodelay, you can use the chart below. This chart shows 7 seconds' worth of accepted and rejected requests for a client sending 20 requests per second to the server:

As we can see, when burst=12, with nodelay, 12 requests are answered in the first second, and in subsequent seconds, 3 requests per second are answered, and once the requests run out, the response rate also drops to zero.

But when delay=9 is used, 3 requests are answered in the first second, and the other 9 requests are placed into the queue. The response rate always stays at 3. Because the requests are placed into the queue, even once the user stops sending new requests, nginx still needs up to 3 more seconds to respond to the requests it put in the queue.

1. With nodelay

   • You can accept up to burst=12 requests instantly, once.

   • As soon as these 12 tokens are used up, the token bucket becomes "empty," and from then on it only adds new tokens at the rate of rate=3r/s.

   • So in the first second, 12 requests are processed instantly, but in the second and third seconds, only 3 each (matching the refill rate) can be processed. You can never send three back-to-back bursts of 12 instant requests unless you wait at least 4 seconds between them for the tokens to refill.

2. With delay

   • No processing happens beyond rate=3r/s.

   • Up to burst=12 of the first requests are "delayed" and go into NGINX's internal queue, but responses still come out at the same 3r/s rate.

   • So those 9 requests that stayed "in the queue" are answered gradually (3 per second), not all 9 at once.

In short:

• The nodelay mode only allows a burst once; after that, refilling happens at the rate.

• The delay mode always responds at the rate, and burst is just the size of the delay queue, not a way to send all responses at once.

Limiting Bandwidth (bandwidth limit)

To create a bandwidth limit, you can use a combination of the limit_rate_after and limit_rate directives.

The bandwidth limit applies per connection. This means if a user has several connections, the limit is calculated separately for each connection.

In the example above, the speed at which the response is served to the client is limited to 1 megabyte per second after 10 megabytes. The bandwidth limit is per connection, so if needed, you can apply a connection limit in addition to the bandwidth limit.

So, based on the settings above, the download speed for the first 10 megabytes is unlimited. After that, the speed is limited to 1 megabyte per second. These settings are usually applied so that users downloading very large files don't hog the server's entire bandwidth, while regular users can still quickly download small files at high speed.

Previous part: Load Balancing | Nginx from Scratch

Next part: Caching | Nginx from Scratch