Reese Knowledgebase

EXT3-fs warning Directory index full!

View Kristian Reese's profile on LinkedIn


If you like this article, please +1 or Recommend via FB with the provided buttons above:

Article ID: 91
by: Reese K.
Posted: 12 Feb, 2013
Last updated: 22 Jul, 2013
Views: 3917

At times, I come across the following message in syslog:

kernel: EXT3-fs warning (device dm-3): ext3_dx_add_entry: Directory index full!


On systems in which this is being constantly logged to syslog, this can become rather annoying, but it is not a bug.  It's informing the user that somewhere within the system, a directory has more than 32,000 subdirectories.  A directory can have at most 31998 subdirectories, because an inode can have at most 32000 links.  (See external link below for source)

Sometimes, this can be difficult to find, especially in a case whereby a physical server is hosting a couple of hundres VPS accounts.  To help break down where the culprit(s) is coming from, I've put together this quick script to tell me where to start looking:

#!/bin/bash
# Kristian Reese
# http://kb.kristianreese.com

for veid in $(vzlist -aHh 0*.tld.com -o veid)
do
  echo $veid
  vzctl exec $veid 'find /var -type d 2>/dev/null |

  (
    while read dir; do cnt=`ls -l "$dir" | wc -l`

    if [ "$cnt" -gt "10000" ]; then
      echo "$cnt: $dir"
    fi

    done
  )'

echo
done

Most of the time, customer emails are queued up in their qmail mailboxes.  I'll investigate to see what these emails are.  For example, in one particular case, I found the customer was running a cronjob to check the status of httpd every waking moment of uptime.  The crontab did not output to /dev/null, and therefore emails were being sent each and everytime the crontjob ran.  This script was named "keep-apache-alive" and was subjected as such in the emails, so I looked to see how many of these there were in the users /var/qmail/mailnames/domain.com/info/Maildir/new.  There were a LOT, so I blew them away and fixed cron to output to /dev/null, preventing the cronjob from sending emails.

-bash-3.1# find . -type f -exec cat {} \; | grep "keep-apache-alive" | wc -l
-bash-3.1# find . -type f -exec grep -q "keep-apache-alive" {} \; -exec rm {} \; -exec echo removed {} \;
-bash-3.1# find . -type f -exec cat {} \; | grep Subject | more

Another method would be to use the following bit of perl code:

#!/usr/bin/perl
use File::Find;
my %dirs = ();
find
(
  sub
  {
   unless(-d $_) { $dirs{$File::Find::dir}++; }
  }, '.'
);

foreach(keys %dirs) {
 if ($dirs{$_} > 20000) {
    print "$_ has $dirs{$_} files.\n";
 }
}

This article was:   Helpful | Not Helpful
External links
http://en.wikipedia.org/wiki/Ext3#cite_note-0

Prev   Next
How to disable cron from sending email alerts     Sort IP Addresses in bash

RSS