NGINX Tuning For Best Performance =================================

For this configuration you can use web server you like, I decided, because I work mostly with it to use nginx.

Generally, properly configured nginx can handle up to 400K to 500K requests per second (clustered). Most what I saw is 50K to 80K (non-clustered) requests per second and 30% CPU load, of course, this was 2 x Intel Xeon with HyperThreading enabled, but it can work without problem on slower machines.

You must understand that this config is used in a testing environment and not in production, so you will need to find a way to implement most of those features as best possible for your servers.

• Stable version NGINX (deb/rpm)
• Mainline version NGINX (deb/rpm)

yum install nginx
apt install nginx

Backup your original configs and you can start reconfigure your configs. You will need to open your nginx.conf at /etc/nginx/nginx.conf with your favorite editor.

# you must set worker processes based on your CPU cores, nginx does not benefit from setting more than that
worker_processes auto; #some last versions calculate it automatically
number of file descriptors used for nginx
the limit for the maximum FDs on the server is usually set by the OS.
if you don't set FD's then OS settings will be used which is by default 2000
workerrlimitnofile 100000;
only log critical errors
error_log /var/log/nginx/error.log crit;
provides the configuration file context in which the directives that affect connection processing are specified.
events {
    # determines how much clients will be served per worker
    # max clients = workerconnections * workerprocesses
    # max clients is also limited by the number of socket connections available on the system (~64k)
    worker_connections 4000;
# optimized to serve many clients with each thread, essential for linux -- for testing environment     use epoll;
# accept as many connections as possible, may flood worker connections if set too low -- for testing environment     multi_accept on; }
http {     # cache informations about FDs, frequently accessed files     # can boost performance, but you need to test those values     openfilecache max=200000 inactive=20s;     openfilecache_valid 30s;     openfilecacheminuses 2;     openfilecache_errors on;
# to boost I/O on HDD we can disable access logs     access_log off;
# copies data between one FD and other from within the kernel     # faster than read() + write()     sendfile on;
# send headers in one piece, it is better than sending them one by one     tcp_nopush on;
# don't buffer data sent, good for small data bursts in real time     # https://brooker.co.za/blog/2024/05/09/nagle.html     # https://news.ycombinator.com/item?id=10608356     #tcp_nodelay on;
# reduce the data that needs to be sent over network -- for testing environment     gzip on;     # gzip_static on;     gzipminlength 10240;     gzipcomplevel 1;     gzip_vary on;     gzip_disable msie6;     gzip_proxied expired no-cache no-store private auth;     gzip_types         # text/html is always compressed by HttpGzipModule         text/css         text/javascript         text/xml         text/plain         text/x-component         application/javascript         application/x-javascript         application/json         application/xml         application/rss+xml         application/atom+xml         font/truetype         font/opentype         application/vnd.ms-fontobject         image/svg+xml;
# allow the server to close connection on non responding client, this will free up memory     resettimedoutconnection on;
# request timed out -- default 60     clientbodytimeout 10;
# if client stop responding, free up memory -- default 60     send_timeout 2;
# server will close connection after this time -- default 75     keepalive_timeout 30;
# number of requests client can make over keep-alive -- for testing environment     keepalive_requests 100000; }

Now you can save the configuration and run the below command

nginx -s reload
/etc/init.d/nginx start|restart

If you wish to test the configuration first you can run

nginx -t
/etc/init.d/nginx configtest

Just For Security Reasons

server_tokens off;

NGINX Simple DDoS Defense

This is far away from a secure DDoS defense but can slow down some small DDoS. This configuration is for a testing environment and you should use your own values.

# limit the number of connections per single IP
limitconnzone $binaryremoteaddr zone=connlimitper_ip:10m;
limit the number of requests for a given session
limitreqzone $binaryremoteaddr zone=reqlimitper_ip:10m rate=5r/s;
zone which we want to limit by upper values, we want limit whole server
server {
    limitconn connlimitperip 10;
    limitreq zone=reqlimitperip burst=10 nodelay;
}
if the request body size is more than the buffer size, then the entire (or partial)
request body is written into a temporary file
clientbodybuffer_size  128k;
buffer size for reading client request header -- for testing environment
clientheaderbuffer_size 3m;
maximum number and size of buffers for large headers to read from client request
largeclientheader_buffers 4 256k;
read timeout for the request body from client -- for testing environment
clientbodytimeout   3m;
how long to wait for the client to send a request header -- for testing environment
clientheadertimeout 3m;

Now you can test the configuration again

nginx -t # /etc/init.d/nginx configtest

nginx -s reload
/etc/init.d/nginx reload|restart

You can test this configuration with tsung and when you are satisfied with the result you can hit Ctrl+C because it can run for hours.

Increase The Maximum Number Of Open Files (nofile limit) – Linux

There are two ways to raise the nofile/max open files/file descriptors/file handles limit for NGINX in RHEL/CentOS 7+. With NGINX running, check the current limit on the master process

$ cat /proc/$(cat /var/run/nginx.pid)/limits | grep open.files Max open files 1024 4096 files

And worker processes

ps --ppid $(cat /var/run/nginx.pid) -o %p|sed '1d'|xargs -I{} cat /proc/{}/limits|grep open.files

Max open files 1024 4096 files Max open files 1024 4096 files

Trying with the workerrlimitnofile directive in {,/usr/local}/etc/nginx/nginx.conf fails as SELinux policy doesn't allow setrlimit. This is shown in /var/log/nginx/error.log

015/07/24 12:46:40 [alert] 12066#0: setrlimit(RLIMIT_NOFILE, 2342) failed (13: Permission denied)

And in /var/log/audit/audit.log

type=AVC msg=audit(1437731200.211:366): avc: denied { setrlimit } for pid=12066 comm="nginx" scontext=systemu:systemr:httpdt:s0 tcontext=systemu:systemr:httpdt:s0 tclass=process

nolimit without Systemd

# /etc/security/limits.conf # /etc/default/nginx (ULIMIT) $ nano /etc/security/limits.d/nginx.conf nginx soft nofile 65536 nginx hard nofile 65536 $ sysctl -p

nolimit with Systemd

$ mkdir -p /etc/systemd/system/nginx.service.d $ nano /etc/systemd/system/nginx.service.d/nginx.conf [Service] LimitNOFILE=30000 $ systemctl daemon-reload $ systemctl restart nginx.service

SELinux boolean httpd_setrlimit to true(1)

This will set fd limits for the worker processes. Leave the workerrlimitnofile directive in {,/usr/local}/etc/nginx/nginx.conf and run the following as root

setsebool -P httpd_setrlimit 1

DoS HTTP/1.1 and above: Range Requests

By default max_ranges is not limited. DoS attacks can create many Range-Requests (Impact on stability I/O).

Socket Sharding in NGINX 1.9.1+ (DragonFly BSD and Linux 3.9+)

| Socket type | Latency (ms) | Latency stdev (ms) | CPU Load | |------------------|--------------|--------------------|----------| | Default | 15.65 | 26.59 | 0.3 | | accept_mutex off | 15.59 | 26.48 | 10 | | reuseport | 12.35 | 3.15 | 0.3 |

Thread Pools in NGINX Boost Performance 9x! (Linux)

Multi-threaded sending of files is currently supported only in Linux. Without sendfilemax_chunk limit, one fast connection may seize the worker process entirely.

Selecting an upstream based on SSL protocol version

map $sslprereadprotocol $upstream {     ""        ssh.example.com:22;     "TLSv1.2" new.example.com:443;     default   tls.example.com:443; }

ssh and https on the same port
server {
    listen      192.168.0.1:443;
    proxy_pass  $upstream;
    ssl_preread on;
}

Happy Hacking! ==============

Reference links

• https://github.com/trimstray/nginx-admins-handbook
• https://github.com/GrrrDog/weird_proxies
• https://github.com/h5bp/server-configs-nginx
• https://github.com/leandromoreira/linux-network-performance-parameters
• https://github.com/nginx-boilerplate/nginx-boilerplate
• https://www.nginx.com/blog/thread-pools-boost-performance-9x/
• https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/
• https://www.nginx.com/blog/nginx-1-13-9-http2-server-push/
• https://www.nginx.com/blog/performing-a-b-testing-nginx-plus/
• https://www.nginx.com/blog/10-tips-for-10x-application-performance/
• https://www.nginx.com/blog/http-keepalives-and-web-performance/
• https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/
• https://www.nginx.com/blog/tcp-load-balancing-udp-load-balancing-nginx-tips-tricks/
• https://www.nginx.com/blog/introducing-cicd-with-nginx-and-nginx-plus/
• https://www.nginx.com/blog/testing-the-performance-of-nginx-and-nginx-plus-web-servers/
• https://www.nginx.com/blog/smart-efficient-byte-range-caching-nginx/
• https://www.nginx.com/blog/nginx-high-performance-caching/
• https://www.nginx.com/resources/wiki/start/topics/examples/x-accel/
• https://nginx.org/r/pcre_jit
• https://nginx.org/r/ssl_engine (openssl engine -t )
• https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus/
• https://www.nginx.com/blog/tuning-nginx/
• https://github.com/intel/asynchmodenginx
• https://openresty.org/download/agentzh-nginx-tutorials-en.html
• https://www.maxcdn.com/blog/nginx-application-performance-optimization/
• https://www.nginx.com/blog/nginx-se-linux-changes-upgrading-rhel-6-6/
• https://medium.freecodecamp.org/a8afdbfde64d
• https://medium.freecodecamp.org/secure-your-web-application-with-these-http-headers-fd66e0367628
• https://gist.github.com/CMCDragonkai/6bfade6431e9ffb7fe88
• https://gist.github.com/denji/9130d1c95e350c58bc50e4b3a9e29bf4
• https://8gwifi.org/docs/nginx-secure.jsp
• http://www.codestance.com/tutorials-archive/nginx-tuning-for-best-performance-255
• https://ospi.fi/blog/centos-7-raise-nofile-limit-for-nginx.html
• https://www.linode.com/docs/websites/nginx/configure-nginx-for-optimized-performance
• https://haydenjames.io/nginx-tuning-tips-tls-ssl-https-ttfb-latency/
• https://gist.github.com/kekru/c09dbab5e78bf76402966b13fa72b9d2

Static analyzers

• https://github.com/MegaManSec/Gixy-Next

• https://github.com/chr4/sslsecure.vim
• https://github.com/chr4/nginx.vim
• https://github.com/nginx/nginx/tree/master/contrib/vim

• https://github.com/rwx------/nginxConfigFormatterGo
• https://github.com/1connect/nginx-config-formatter
• https://github.com/lovette/nginx-tools/tree/master/nginx-minify-conf

• https://github.com/nginxinc/crossplane
• https://github.com/valentinxxx/nginxconfig.io

• https://blog.cloudflare.com/http-2-prioritization-with-nginx/
• Linux v4.13+ as no longer required FQ (q_disc) with BBR.
• https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md
• https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=218af599fa635b107cfe10acf3249c4dfe5e4123
• https://github.com/systemd/systemd/issues/9725#issuecomment-413369212
• If the latest Linux kernel distribution does not have tcp_bbr enabled by default:

modprobe tcpbbr && echo 'tcpbbr' >> /etc/modules-load.d/bbr.conf echo 'net.ipv4.tcpcongestioncontrol=bbr' >> /etc/sysctl.d/99-bbr.conf 
Recommended for production, but with  Linux v4.13rc1+ can be used not only in FQ (`q_disc') in BBR mode.
 echo 'net.core.default_qdisc=fq' >> /etc/sysctl.d/99-bbr.conf sysctl --system