Home > Linux, Security > Snort: too many open files

Snort: too many open files

Creation of insane rule
All we hear:
Desperate cry
(Sepultura – Desperate Cry)

I really hate those unproductive hours (hopefully not days) when one needs to debug some strange problems whose solution won’t be reusable. Hm? Deja-vu? 🙂 Well it hit me again. And this time it was hard.

I was trying to write some manifests and control our local Snort installation through puppet. We use VRT and emerging rules, fetched via pulledpork. So, puppetizing Snort should be like a breeze. And it was… Everything went extremely well, I wrote two classes: snort and snort::pulledpork (along with standard params class). Data was stored in hiera, /etc/sysconfig/snort, /etc/snort/snort.conf and all of the pulledpork configs are dynamicly generated from that data. World looked really nice. And I was a happy devop 😉

But the problems started later – when I actually tried to start snort service. Service was just failing miserably without any significant output. I’ve tried ‘bash -x’ on the init script, and running manual command, but I was getting nowhere. Then I turned to syslog, and I saw a bunch of Snort startup messages and then all of a sudden:

rsyslogd-2177: imuxsock begins to drop messages from pid 17207 due to rate-limiting

Well, temporary fix for that issue was:

# echo "$SystemLogRateLimitInterval 0" > /etc/rsyslog.d/test.conf

And offcourse, you need to restart rsyslogd after this one… Pretty strange that default syslog in CentOS 6 is so itchy about being filled up too fast… I did like old syslogd behaviour more…

Anyways, back to the main issue. After “fixing” rsyslogd I finally had something to work on:

FATAL ERROR: /etc/snort/rules/VRT-app-detect.rules(0)
 Unable to open rules file "/etc/snort/rules/VRT-app-detect.rules":
 Too many open files.#012

Now we’re getting there! It’s a piece of cake to solve:

# echo "ulimit -n 10240" >> /etc/sysconfig/snortd

Although it didn’t work… So off I was on a lonely path of useless debugging. Why doesn’t this work? Maybe it’s something with the system? Trying to increase fs.file-max to absurd levels didn’t help…. Maybe it’s to do with account snort will run as – snortd? Trying to utilize limits.conf didn’t work either. Now I was buffled. One thing I did notice was that after raising ulimit on number of open files, snort was starting, or should I say failing, a lot longer… Then I decided to utilise strace. Number in the “read” system call was just raising and raising until hitting the maximum. The weird thing was that it always broke on the exact same file… That drag me away of real problem. So nothing helped so far – so I decided to dismantle snort configurations and rules. And after zeroing out one config file – snort started! Now we’re talking. I decided to uncomment line by line. After 3/4 of lines, another error… And now I finally saw the culprit!!!

include $RULE_PATH/VRT.conf

Utter facepalm… I’ll leave you to guess the name of the file that contained that line…

Advertisements
  1. April 2, 2013 at 10:10 am

    Nice article! did you share your puppet files anywhere?
    Paolo

    • April 6, 2013 at 12:07 am

      Hi. I have part of Puppet modules for managing Snort online, although they are not yet ready for publishing on forge… If you can contact me privately I will send you link to download the module.

      • April 8, 2013 at 8:57 am

        How can I contact you privately? cannot find your email address on the blog anywhere

  2. February 6, 2016 at 9:58 pm

    I don’t understand, but I’m pretty dense. I also have never used snort and I feel like there’s some sort of Snort inside joke going on here.

    Was the culprit that the $RULE_PATH/VRT.conf file was calling itself? (i.e. that line was recursively calling itself)

    Of course, this is from 3 1/2 years ago so perhaps you no longer remember.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: