Fix MariaDB/MySQL Won't Start: InnoDB Crash Recovery

You run systemctl restart mariadb and get a red "failed" back. Every WordPress site on the box now shows "Error establishing a database connection", and your phone is lighting up. This usually happens after an out-of-memory kill, a full disk, or a hard reboot, and in most cases your data is fine. MySQL or MariaDB just needs a little help getting through InnoDB crash recovery. Here's how we work through it on a VPS.

Symptoms: how this failure shows up

• systemctl status mariadb (or mysqld) shows failed or keeps flipping to activating.
• Sites show "Error establishing a database connection" while the web server itself is healthy.
• The service starts, runs for a few seconds, then dies again.
• mysql says Can't connect to local MySQL server through socket.

Step 1: Read the real error before touching anything

Don't guess. The log tells you which of the causes below you have.

systemctl status mariadb -l --no-pager
journalctl -u mariadb -n 80 --no-pager
tail -n 100 /var/log/mysql/error.log      # Debian/Ubuntu
tail -n 100 /var/lib/mysql/*.err          # AlmaLinux/Rocky/cPanel

On cPanel servers the log is usually /var/lib/mysql/hostname.err. Look for the first line containing ERROR or [Note] InnoDB: ... crash recovery, not the last one. The last lines are usually just the shutdown noise.

Step 2: Match the log line to a cause

Fix A: The disk is full

MariaDB can't write its redo log or temp files, so it refuses to start. Check both space and inodes:

df -h /var/lib/mysql
df -i /var/lib/mysql
du -sh /var/lib/mysql/* | sort -h | tail

Free a few hundred MB first. Old binary logs (mysql-bin.*), huge error_log files and old backups are the usual culprits. Then start the service again. Don't delete anything inside database folders by hand.

Fix B: A stale or duplicate mysqld process

If the log mentions a lock error, something already has the data files open.

ps aux | grep -E 'mysqld|mariadbd' | grep -v grep
ss -ltnp | grep 3306

If an orphaned process is hanging around after a failed stop, end it gracefully with kill -TERM <pid>, wait about 30 seconds (it may be flushing), and only then consider kill -9. Start the service afterward.

Fix C: Out of memory

This is the most common cause on small VPS plans. The kernel's OOM killer ended MariaDB, and now it can't reserve what it wants at startup.

dmesg -T | grep -i -E 'out of memory|killed process' | tail
free -m
grep -i innodb_buffer_pool_size /etc/my.cnf /etc/my.cnf.d/* /etc/mysql/*.cnf /etc/mysql/*/*.cnf 2>/dev/null

As a rule of thumb, keep the buffer pool at roughly 40-50% of RAM on a shared server (a 2 GB VPS running PHP too should be closer to 512M-768M). Lower it, add a swap file if you don't have one, then restart. If the OOM killer keeps visiting, the real fix is fewer PHP workers or a bigger plan. Our post on OOM killer and swap tuning covers that side.

Fix D: InnoDB corruption and forced recovery

If the log shows corruption or an assertion failure, back up first. Copy the whole data directory before any recovery attempt:

systemctl stop mariadb
cp -a /var/lib/mysql /root/mysql-datadir-backup-$(date +%F)

Then add a recovery level to the config under [mysqld], starting at 1:

[mysqld]
innodb_force_recovery = 1

Restart. If it still fails, go to 2, then 3. Stay at 3 or below if you can. Levels 4 to 6 can permanently lose data and are a last resort.

Once it's up in forced-recovery mode, the server is read-only in practice. Don't try to run the sites. Dump everything out right away:

mysqldump --all-databases --routines --events > /root/all-dbs-$(date +%F).sql

Then stop MariaDB, remove the innodb_force_recovery line, move the damaged data directory aside, initialise a fresh one, and import the dump. This is slower than the fix you were hoping for, but it's the only clean way to end up with a healthy server.

Fix E: Ownership and SELinux

If someone restored files as root, or moved the data directory, MariaDB can't read its own files.

ls -ld /var/lib/mysql
chown -R mysql:mysql /var/lib/mysql
restorecon -Rv /var/lib/mysql     # only on SELinux systems

Fix F: A bad config change

If it broke right after you edited my.cnf, the log will name the offending variable. Comment out your last change, restart, and reapply settings one at a time. Check the syntax with mysqld --help --verbose 2>&1 | head to catch typos early.

Verify the repair

systemctl status mariadb --no-pager
mysqladmin status
mysqlcheck --all-databases --check

If mysqlcheck reports tables as corrupt, repair them from the dump rather than relying on REPAIR TABLE, which doesn't apply to InnoDB anyway. On cPanel/WHM, you can also run /scripts/restartsrv_mysql and watch the output.

Prevention

• Alert on disk space at 80%, and rotate binary logs with expire_logs_days or binlog_expire_logs_seconds.
• Size the buffer pool honestly. Leave room for PHP-FPM, Nginx and the OS.
• Add swap as a safety net, even a 2 GB file.
• Never power-cycle a VPS without a clean systemctl stop mariadb if you can avoid it.
• Take nightly dumps and a snapshot before big changes. A recovery from a good backup beats any forced-recovery trick.

Frequently Asked Questions

Will I lose data if MySQL crashed?

Usually not. InnoDB replays its redo log on startup and rolls back unfinished transactions. Data loss mostly comes from real disk corruption or from using high innodb_force_recovery levels.

Is it safe to delete ib_logfile0 and ib_logfile1?

Only after a clean shutdown, and only as a deliberate step. Deleting them after a crash throws away the very log InnoDB needs for recovery. Back up the data directory first.

MariaDB starts, then dies after a minute. Why?

That pattern points to memory pressure or a corrupted table that's touched by a background thread. Check dmesg for OOM kills and the error log for assertion failures.

Can SkyServer support help recover a crashed database?

Yes. Open a ticket with the error log output and tell us whether you have a snapshot or backup. We'll take a look before you try anything above level 3 of forced recovery.