open file limits for nginx and rabbitmq scale
OS
There are separate limits for different users, so make sure to run this as the user your process is using if you want to check.
$ ulimit -n 1024
There is a soft limit and a hard limit.
The soft limit is the actual limit your processes have to obey.
$ ulimit -Sn 1024
The hard limit is the maximum number the soft limit can be set to.
$ ulimit -Hn 1024
To set the open file limit for the current active shell session (lost on logout)
$ ulimit -n 4096 $ ulimit -n 4096
To set the open file limit permanently
$ vim /etc/security/limits.conf * soft nofile 4096 * hard nofile 4096 -- or for a specific user -- root soft nofile 4096 root hard nofile 4096
You need to then reboot the machine for it to take effect.
$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 15720 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 4096 <------------------- we changed this one pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 15720 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
NGINX
There are three options you can focus on here.
worker_processes = should be equal to the number of cpu cores available on the machine worker_connections = important for the number of concurrent active connections, default is 1024 worker_rlimit_nofile = represents the number of file handles for each connection, x1 for a file server, x2 if the connection is being proxied, ie, one in and one out eg, if you expect to proxy 100 connections, this number would need to be 200
These options should be set in the main nginx config file.
$ vim /etc/nginx/nginx.conf . . worker_processes auto; . . events { worker_connections 4096; . . } worker_rlimit_nofile 4096; http { . . . }
Check the nginx process's limits to see that they reflect the changes on the os.
$ ps -ef | grep nginx root 1556 1 0 Jan04 ? 00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; www-data 1557 1556 0 Jan04 ? 00:00:08 nginx: worker process www-data 1558 1556 0 Jan04 ? 00:00:09 nginx: worker process www-data 1560 1556 0 Jan04 ? 00:00:00 nginx: cache manager process root 61143 60933 0 15:15 pts/0 00:00:00 grep --color=auto nginx
The worker processes are the important ones to check as...
$ cat /proc/1557/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15720 15720 processes Max open files 4096 4096 files <-------------------- Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15720 15720 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
...the master process is not the one opening and closing file handles, but managing the worker processes
$ cat /proc/1556/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15720 15720 processes Max open files 1024 4096 files <-------------------- Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15720 15720 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
RABBITMQ
In addition to the os nolimit value, we also need to add this:
$ vim /etc/default/rabbitmq-server ulimit -n 512000
Check the rabbitmq process limits as well.
$ ps -ef | grep rabbit rabbitmq 2002 1 0 Jan04 ? 00:00:00 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon rabbitmq 2037 1 0 Jan04 ? 00:00:00 /bin/sh -e /usr/lib/rabbitmq/bin/rabbitmq-server rabbitmq 2358 2037 1 Jan04 ? 00:25:56 /usr/lib/erlang/erts-5.10.4/bin/beam -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl ... rabbitmq 2742 2358 0 Jan04 ? 00:00:01 inet_gethost 4 rabbitmq 2743 2742 0 Jan04 ? 00:00:01 inet_gethost 4 rabbitmq 5070 2742 0 Jan04 ? 00:00:00 inet_gethost 4 root 38633 38610 0 15:18 pts/0 00:00:00 grep --color=auto rabbit $ cat /proc/2002/limits $ cat /proc/2037/limits $ cat /proc/2358/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 7656 7656 processes Max open files 512000 512000 files <-------------------- Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 7656 7656 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
