Steam on Linux: Many games won't start? Read this! [If you use XFS on a 1TB+ HDD]

[CodAv]

Dear fellow Linux users,

some of you might have run into the same issue I did: many games, mostly Source and Unity3D games, don't start at all or come up with weird error messages like "file XY not found". It took me some time to sort out the cause of the issue, and I want to share my findings with the community. First of all, if the following aspects fit your system configuration, you'll be most probably affected.

Affected system configuration

• You're running a 64 bit Linux Distribution with a 32 bit compatibility layer (Multilib/Multiarch)
  
• You've got a hard disk with a size of more than 1 TB on the partition you're using for Steam
  
• You're using a file system which uses 64 bit inodes (most probably XFS) on that partition

Some affected games

• Among the Sleep
  
• Anna Extended Edition
  
• Awesomenauts
  
• Counter-Strike Source
  
• Crusader Kings II
  
• Half Life 2
  
• Left4Dead 2
  
• Portal 2
  
• Sid Meier's Civilization V
  
• Team Fortress 2

Solution / Workaround

The most simple and easiest solution to the problem is to install your games on a hard disk using a filesystem which always works with 32 bit inodes, e.g. ext4. You can limit this to non-working games, as 64-bit titles and correctly compiled games are not affected.

Using the "inode64" option to mount XFS will not solve the issue.

Other workarounds are going deeper into your system, telling your kernel to wrap 64 bit inode numbers down to 32 bit, or creating a small extra library which uses the 64 bit filesystem functions, and wraps the return values and structures down to the 32 bit variants, which are injected to the application using the LD_PRELOAD environment variable. These solutions might work, but can cause REAL trouble like overwriting other files in your partition.

The best solution would be to compile the offending application using some special compiler definitions (see technical explanation below), but this is up to the developers, and in case of the Unity3D player, even requires a recompilation of the 3rd party software.

Why is this actually causing my games to crash?

Warning: lengthy technobabble!
In most cases, file system access is abstracted by the system kernel and the operating systems standard C library (glibc on most Linux distributions). So applications don't have to know which file system or what kind of hard drive you're using, they just call a set of functions to read, write or get information about files. Historically, these functions work with 32 bit numeric values, which limits the maximum size of a file which can be accessed to about 4 GB. Some other limits apply, but in most scenarios, these limits are scarcely scratched. But there are some functions which work on files using the filesystem-specifc file number, called "inode". Most UNIX filesystems store inodes with 32 bit numbers, which means you can have up to 4.2 billion files in a single filesystem, which should be enough in most cases. But some filesystems are using these numbers differently, or support very large filesystems up to exabyte ranges where the number of inodes could go above this limit. XFS is one of those, on partitions larger than 1 TB, it uses 64 bit inodes.

If you've got a 32 bit application compiled with no special definitions, the C library IO functions return inode numbers, file offsets and sizes as unsigned 32 bit numbers. If the kernel puts a 64 bit number into some of these values, the data structure will get corrupted, either by overlapping fields (one half of a number filling the next field in the structure) or causing an overflow, the numbers are wrong. Depending on what the application uses these values for, many different errors can occur, e.g.:

• A file or folder can't be found
  
• Creating a new file fails
  
• The application might use the file size reported by stat() to allocate memory

To overcome the issue, developers have to use the filesystem-specific types defined in the standard library (size_t, off_t, ino_t etc.), make sure these values are not cast to int or long values, and then provide the two extra glibc macro definitions "_FILE_OFFSET_BITS=64" and "_LARGEFILE64_SOURCE" while compiling each source file.

[[BPR] Lady]

all games start normal, so this is a pointless thread.

[Zyro]

Originally posted by REB Lady From Hell:

all games start normal, so this is a pointless thread.

For everybody or for you?

I didn't yet use partitions that large, so I didn't meet the problem either,
but thanks to the OP for sharing his findings!

[CodAv]

Originally posted by REB Lady From Hell:

all games start normal, so this is a pointless thread.

Well, for, let's say, more than 99% of all Linux users, this thread is indeed pointless, as the described system configuration is far off from standard, as most distributions use ext4 or btrfs by default, and both aren't using 64 bit inode numbers. But for affected people, this might save a lot of time and frustration. Not everyone reads through Valve's GitHub bugtracker, understanding the quite technical discussions. Even there, it took people several months to sort out the real cause.

[Dusk of Oolacile]

Originally posted by CodAv:

the described system configuration is far off from standard

Indeed it is. I think Steam is for the casual gamer. You also can't run games if you set up a very strange Windows configuration.

[Zyro]

You might mention something in the title like "... for large HDs with unusual file systems" or something along the lines.

[snowboarder74]

Hey bud, thanks so much for this!! I can't start over half of my games and I just couldn't figure out why. Thanks to your post from 3 yrs ago the answer has been illuminated!!

I run the steam client from workstation1 but my steam folder with all my games sits on a separate 20TB XFS raid0+1 connected to workstation 2 which communicates via NFS to all other workstations.
Some games work and others just flash for a second or I get a warning about steam.so

To everyone who was saying that this post is useless, it may be useless to you but eventually someone like me comes along and has his problem answered.

[TOBIAS N.]

Some Games have dependencies out of the intern Steam-own Linux Library bunch, so they must be solved within your Linux-Distribution, this is mostly the 32Bit compat Pack of your Linux, but also 64Bit Libraries must be solved if the Game is 64Bit.

- A simple way to see what's missing, is to start the client from a shell, then Start the Game out of the Client and look what happens in the text messages that come up......!

[puleglot]

Originally posted by CodAv:

Affected system configuration

• You're running a 64 bit Linux Distribution with a 32 bit compatibility layer (Multilib/Multiarch)
  
• You've got a hard disk with a size of more than 1 TB on the partition you're using for Steam
  
• You're using a file system which uses 64 bit inodes (most probably XFS) on that partition

mkfs.xfs from recent xfsprogs creates filesystems with V5 metadata and with 512-byte inodes by default. Such filesystems actually start using 64-bit inodes when their size is >2 Tb. See [1] for reference.

~ $ sudo mkfs.xfs -f /dev/mapper/vg_system-testlv 2>&1 | egrep -o 'isize=[^ ]*'
isize=512
~ $ sudo mkfs.xfs -f -m crc=0 /dev/mapper/vg_system-testlv 2>&1 | egrep -o 'isize=[^ ]*'
isize=256

[1] http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/ch06s06.html

[CodAv]

Originally posted by Zyro:

You might mention something in the title like "... for large HDs with unusual file systems" or something along the lines.

Added a hint to the title.

@puleglot: Thanks for the details! My HDD just happens to have a size of 2TB.