Reese Knowledgebase

[ERROR] /usr/libexec/mysqld: Incorrect information in file

View Kristian Reese's profile on LinkedIn


If you like this article, please +1 or Recommend via FB with the provided buttons above:

Article ID: 119
by: Reese K.
Posted: 29 Jul, 2013
Last updated: 30 Jul, 2013
Views: 3588

While supporting a customer account, I came across this mysql issue:

[ERROR] /usr/libexec/mysqld: Incorrect information in file  './drupal_b/semaphore.frm'

This error repeated many times over for each frm file for practically every database in existence on this system.  I dug further, looking at mysql error log in /var/log/mysqld.log and came across some  helpful messages:

InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.

For this message, I could see that the customer had enabled mysqld service in run level 3 which was unnecessary because the application also starts mysql.  Simply removing the mysqld service via chkconfig corrected this error.


130729 11:43:33  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 1 3408329469

After letting mysql run for a long period of time, the so called recovery never seemed to complete or do much of anything other than spin up CPU to near 100% utilization.  A subsequent strace of the process revealed that it wasn't doing anything, so I began looking into other ways to get services restored.  Taking information from the log file, it was apparent something was up with the ib data files in /var/lib/mysql.  I took backups of everything and began playing around different scenarios.  The combination that ended up working for me was to:

  1. rm -f ib_logfile{0,1}
  2. /etc/init.d/mysql restart

This allowd mysql to start without apparent error from command line output, which was a good sign even though the error log still indicated there was an issue, logging the following:

130729 17:11:36  InnoDB: Error: page 539 log sequence number 1 3058776649
InnoDB: is in the future! Current system log sequence number 0 76934.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
InnoDB: for more information.

Regardless, I now had a mysql prompt to work with and could now query the table space without getting the "Incorrection information" error.  Now I know I can take good dumps and recover the Innodb database engine that is corrupted.  The process involves:

  1. dump all databases
  2. remove the databases and corrupted InnoDB files
  3. recreate a base mysql engine
  4. restore databases from dump

To begin, stop mysqld and backup all database files:

~# /etc/init.d/mysql stop
~# mkdir /root/mysql_backups
~# cp -r /var/lib/mysql/* /root/mysql_backups

Now we'll want to force innodb recovery by adding the following entry into the [mysqld] section of the  MySQL configuration (in my case, this file is /etc/my.cnf) and star

[mysqld]
 innodb_force_recovery = 4

Start mysql:
/etc/init.d/mysqld start

Make a dump of all mysql databases:
~# mysqldump -uadmin -p -A > /root/mysql_backups/dump.sql

Tip: If the password isn't working, add 'skip-grant-tables' to /etc/my.cnf, restart mysql, then try the dump again

Once the dump completes, the process is ready to recreate the main database "mysql" and the InnoDB engine.  This is simply performed by starting mysql.  Once complete, restore all backups from the dump taken earlier:

~# /etc/init.d/mysqld stop
~# rm -rf /var/lib/mysql/*
<edit /etc/my.cnf and REMOVE innodb_force_recovery line>
~# /etc/init.d/mysqld start
~# mysql -uadmin -p < /root/mysql_backups/dump.sql


There is an interesting article discussing InnoDB recovery times in the external links section below.

This article was:   Helpful | Not Helpful
External links
http://www.mysqlperformanceblog.com/2007/05/09/how-to-estimate-time-it-takes-innodb-to-recover/

Also listed in
folder VPS

Prev   Next
SQL     Watch active MySQL queries as they come and go

Showing: 1-1 of 1  
Comments
Tom Fielding | 08 Oct, 2014 01:07 PM
MySQL wouldn't start after removing the contents of /var/lib/mysql. I had to do some changes, so I figured in case anyone else runs into this, I'd put up what I did.
First I had to raise the InnoDB buffer size (it was 4MB, I set it to 32M temp to get it to work).
/etc/my.cnf innodb_buffer_pool_size=32M

Then I got MySQL to start but had to set the MySQL passwords
/usr/bin/mysqladmin -u root password PASSWORDHERE
/usr/bin/mysqladmin -u root -h ******.host.com password PASSWORDHERE

After doing this I was able to restore the MySQL dump. I restarted the entire server after clearing /var/log/mysql.log and verified no errors (a healthy shutdown and startup).

I then made another DB backup for good measure.
rm -rf /root/mysql_backups/*
cp -r /var/lib/mysql/* /root/mysql_backups
mysqldump -uadmin -p -A > /root/mysql_backups/dump.sql
Prev   Next
SQL     Watch active MySQL queries as they come and go

RSS