Increase RabbitMQ file descriptor limit and memory watermark without restart

Our time here
Is now at end
Can’t help but reminisce
A cold spring day
So long ago
When we set out to sea
(Amon Amarth – Varyags Of Miklagaard)

In this all-demanding world, there is often a need to change some security (or other) limits of running processes without restarting them. One situation I found my self in was need to increase RabbitMQ’s file descriptor limit. By default, for non-root processes limit is 1024. Lets describe the process when you want to change limits with restart first.

When you want to increase that limit, first thing to change would be the settings of security limits. We can do that by creating new file in /etc/security/limits.d/. Lets name it rabbitmq.conf:

# cat >> /etc/security/limits.d/rabbitmq.conf <<EOF
# rabbitmq
# Increase maximum number of open files from 1024 to 4096 for RabbitMQ

#<domain>    <type>    <item>    <value>
rabbitmq    soft    nofile    4096
EOF

To check if this setting is correct, run:This is to ensure Now, lets create directories for git and create our first repo:

# su - rabbitmq -s /bin/sh -c 'ulimit -n'
4096

Now, it would be enough to restart RabbitMQ and new settings would get detected. Unfortunately if you don’t want to or can’t restart Rabbit, we have to do some more mingling.

First, we need to find out all processes running as user RabbitMQ:

# ps auxw | grep ^rabbit | cut -d' ' -f 3
2103
6300
7204
7334
7335

Now we can check for each of those PIDs what are their current limits, for example for PID 2103:

# cat /proc/2103/limits  | grep open
Max open files            4096                 4096                 files

Now for each of these processes we need to increase ulimit. We can do that by running:

# echo -n "Max open files=4096:4096" > /proc/2103/limits

After this, limits will be increased and processes should know that. But looking at RabbitMQ status output, we can see that it’s not the case:

# rabbitmqctl status | grep -A 4 file_descriptors
 {file_descriptors,
     [{total_limit,924},
      {total_used,291},
      {sockets_limit,829},
      {sockets_used,195}]},

Now, to increase the value, just run the following command. Note that we increase it to ulimit minus 100, so it’s not 4096 but 3996. That’s the way RabbitMQ does it by default so we will stick to it:

# rabbitmqctl eval 'file_handle_cache:set_limit(3996)'

If you, on the other hand, just want to increase memory usage without restarting RabbitMQ, that can be done with simple:

# rabbitmqctl set_vm_memory_high_watermark 0.8

Also, don’t forget to save vm_memory_high_watermark settings in rabbitmq.config file, so that they are persistent across (eventual) restarts.

Note: If you run CentOS 7 or Fedora which use systemd, it’s not enough to change ulimits via limits.conf, you need to actually set “LimitNOFILE=32768” in .service file. Hip hip hurray for systemd (sigh).

Note2: Thanx to Simon MacMullen for his help on the issue.