Skip to main content
Thoughts from the DevOps world

XFS Corruption: Repairing

2 min read Dan linux scripting

Note: The go-to tool for XFS repairs is, unsurprisingly, xfs_repair. Getting it to run, however, can sometimes be the hard part.

Introduction

How does filesystem corruption happen? There are several likely causes:

  • Kernel bugs: Rare, but they have happened many times in the past and will happen again. Little can be done beyond applying patches and keeping the kernel up to date.

  • Memory issues: For example, memory errors that propagate into the filesystem’s control structures. ECC memory mitigates this, but it can never be fully ruled out.

  • Underlying storage problems: Quite unlikely, but still possible.

  • Hitting the reset button on a running server: Journaling filesystems can almost always recover from this.

  • RAID controller issues: Possibly the leading cause, and not easy to mitigate, even though a firmware upgrade is sometimes available.

Any filesystem type can be affected by any of these causes. XFS simply appears more often in corruption-related search results because it is so widely used.

Disaster?

You wake up in the morning, happy, with the whole day planned out. Then an error pops up on your pager:

“Corruption of in-memory data detected. Shutting down filesystem”

This one is bad. If the filesystem holds databases or web server files, things will go south within minutes: services will go down or stop serving users. That is why smart people replicate their services, use load balancers, or run a hot-replica architecture.

Repairing

The only complicated part is that you need to be on the server’s real keyboard and video (for example via a KVM switch). That may mean a trip to the server’s physical location.

Assuming that requirement is met, the repair itself is straightforward:

  1. Boot into single-user mode. For example, edit the kernel line in GRUB and add the keyword single at the very end. This gives you a root console with no password prompt and no services running.

  2. Unmount the corrupted filesystem using umount.

  3. Run xfs_repair against that filesystem.

Minutes later, since xfs_repair is usually quite fast, a normal reboot should bring the system back to its former glory. Some files may be missing, and some content or database tables may be corrupted, but that’s life. At least everything starts up normally again for now.

Conclusion

Easy? It is. Well, until the next time…