[Linux] Reproducible crash at first video cutscene

[Phatty]

Since playing the 1.0 release, I started a new game and I can't get past the first video cutscene due to memory exhaustion. I've tried DX11, DX12, Proton Experimental, Proton 7, 6, various GE releases - none help. I've tried DLSS, FSR, lowering textures... nothing helps.

It's completely playable (and performs well too) right up until the first cutscene after you boost to Ben. Once it switches to the video cutscene where you're imprisoned, I can see my VRAM start filling (~5GB in use under normal play, 3080 10GB), then once that caps out, it starts consuming system memory. It will do this until it fills that up (32GB), then it will produce an error about allocating memory, then it crashes.

This happens every single time, so I currently can't play at all

Please let me know if I can provide any information to help debug this as I've been looking forward to playing this over the Easter weekend

OS: Pop_OS 22.04
NVIDIA Driver Version: 525.89.02

EDIT: Looking at the CrashContext.runtime-xml, it shows some very interesting memory stats:

<MemoryStats.TotalPhysical>33550336000</MemoryStats.TotalPhysical>
<MemoryStats.TotalVirtual>140737488224256</MemoryStats.TotalVirtual>
<MemoryStats.PageSize>4096</MemoryStats.PageSize>
<MemoryStats.TotalPhysicalGB>32</MemoryStats.TotalPhysicalGB>
<MemoryStats.AvailablePhysical>1461989376</MemoryStats.AvailablePhysical>
<MemoryStats.AvailableVirtual>21007962112</MemoryStats.AvailableVirtual>
<MemoryStats.UsedPhysical>594272256</MemoryStats.UsedPhysical>
<MemoryStats.PeakUsedPhysical>3984257024</MemoryStats.PeakUsedPhysical>
<MemoryStats.UsedVirtual>3133784064</MemoryStats.UsedVirtual>
<MemoryStats.PeakUsedVirtual>3133784064</MemoryStats.PeakUsedVirtual>
<MemoryStats.bIsOOM>0</MemoryStats.bIsOOM>
<MemoryStats.OOMAllocationSize>0</MemoryStats.OOMAllocationSize>
<MemoryStats.OOMAllocationAlignment>0</MemoryStats.OOMAllocationAlignment>

I'm pretty sure I don't have 140 _terabytes_ of virtual memory.

This is the only thread showing as crashed:
<Thread>
<CallStack>ntdll 0x0000000170000000 + e294 kernelbase 0x000000007b000000 + 6d3aa kernelbase 0x000000007b000000 + 6dace ES2-Win64-Shipping 0x000000014
0000000 + 1a2da90 ES2-Win64-Shipping 0x0000000140000000 + 416e7f2 VCRUNTIME140 0x0000000001430000 + c1c0 ntdll 0x0000000170000000 + 58a84 ntdll 0x0000000170000000 + 5a96b ntdll 0x00
00000170000000 + 5b0a9 ntdll 0x0000000170000000 + 58c54 kernelbase 0x000000007b000000 + 1260e ES2-Win64-Shipping 0x0000000140000000 + 1a2da16 ES2-Win64-Shipping 0x0000000140000000 +
19eee6d ES2-Win64-Shipping 0x0000000140000000 + 1eb4b48 ES2-Win64-Shipping 0x0000000140000000 + 1eb2173 ES2-Win64-Shipping 0x0000000140000000 + 1e85d30 ES2-Win64-Shipping 0x0000000
140000000 + 1ea8966 ES2-Win64-Shipping 0x0000000140000000 + 1285581 ES2-Win64-Shipping 0x0000000140000000 + 128777a ES2-Win64-Shipping 0x0000000140000000 + 18f38cb</CallStack>
<IsCrashed>true</IsCrashed>
<Registers />
<ThreadID>776</ThreadID>
<ThreadName>RenderThread 1</ThreadName>
</Thread>

[Staunch]

I hit this as well and I know it's not a great solution but I was able to get past it by hitting escape and skipping the cutscene.

[Staunch]

For what it's worth, it looks like the first cutscene is unchanged from the first 3 minutes here if you want to still see the story: https://youtu.be/FgWEN1NBMO0

[sudo rm -rf /]

Got this as well, good to see it's not only me!
I haven't bothered skipping past, so I don't know if it happens on other cut scenes as well?
(Kind of want to follow the story and get immersed for now, so don't really want to skip it)

[Staunch]

You unlock the ability to replay videos very early into the game, after the tutorial and first area where you jump away. There's a data menu when you're at your homebase which has a "The Story So Far" section that lets you reply videos. From my quick test it seems playing the videos from there doesn't exhibit the memory problem.

[Jarmer]

I can confirm what @staunch said - I'm able to skip the scenes, then rewatch without issue. I have not tracked my gpu during the game to see if this is vram overflow or not, but when I play it later today I will and report back.

[Phatty]

It's still an issue with the replay, but to a lesser extent. Cutscene 01, 03 and 04 work fine, but 06 consumes all the VRAM then starts gobbling up the system RAM, although it stopped at about 28GB used and then went down again. There's definitely an issue with the cutscene stuff.

[KiasuBurger]

Now you have a use for a 24GB GPU :-)

Has anyone tried with GE-Proton? Playing the original video files rather than the Valve transcoded ones might have different behaviour.

[Staunch]

I just tested with the latest GE-Proton and it still exhibits the increasing memory usage in the first cutscene.

As another workaround if people are looking for one, the video files are all accessible in ES2/Content/Movies in the game folder (right click on the game in Steam and click Manage > Browse Local Files). If you can find the one you want to watch by checking the first few seconds from what you saw in game you can use a player that's able to open a separate video and audio files like VLC to play the cinematics.

[Jarmer]

I tried with GE Proton as well - no change. Still crashes in the cutscenes. I have to be QUICK on the F button to skip them as fast as possible otherwise it locks up and crash. I haven't gotten far enough to test the other ones.

[KiasuBurger]

I have a 4090 here so I can actually play Video 06 to completion, but it absolutely maxes out the 24GB. One interesting observation is that the memory usage falls off as the clip nears the end, which suggests that this isn't a simple resource leak (eg: every frame being loaded into memory and then never released until the end of playback) but something more like a caching behaviour where the size management isn't working as intended.

[PocketSquirrl]

For me it worked to lock the fps to 60. Not sure how much memory it consumed then.

[astromedia.only]

Same on SteamDeck it seems (on every longer cutscene so far).

[blackcoffeerider]

I also started testing all sorts of combinations with proton and also downgraded kernel and nvidia drivers but it does not seem to make a difference.

In general if you do run into this problem and do some testing: please be aware that if run your swap space on an SSD this will cause unnecessary writes.
So as a safety measure either do SWAPOFF when testing or look into something like https://github.com/rfjakob/earlyoom

I am trying to gather as much information as i can but i am not sure if it realy helps posting inxii and journalctrl stuff here randomly...


Unfortunately i also dont seem to have a CrashContext.runtime-xml - is there a flag that i have to switch on?

Edit 1:
I found some realy old Crash Reports in the library folder where the game is installed: compatdata/1128920/pfx/drive_c/users/steamuser/AppData/Local/ES2/Saved/Crashes ?

[blackcoffeerider]

I might have found a workaround by reencoding the videos to a very small format.
I ended up using VP8 in 720x405 at 10 FPS
It is not nice but imho better than trying to skip and searching the cutscene online to watch.

I am aware that the cutscene that i am stuck at is by far not the longest in the game so this might just be a bandaid for some hours into the game :(
The limiting factor is video ram - it the system runs out of that it startt to move VRAM to system memory where the stuttering starts.

Relevant System info:
Proton version: 7.0-6e
Memory:
VRAM: 8192 MB
RAM: 32008 MB

When entering the cutscene the vram is interestingly enough filling up fast, then leveling off around 6000MB and then near the last third of the movie falling off again

--
I am assuming here that you know some basics in the terminal ;)

0) go to the crimescene
open the local files directory (in steam right click everspace2 > properties > local files > browse)
in that find the sub-directory
ES2/Content/Movies

1) make a backup

mkdir webmorg
mv *.webm webmorg/
cp BatchConvertMp4.sh BatchConvertMp4.org

2) modify the conversion script

sed -i 's/\r$//' BatchConvertMp4.sh
sed -i 's/libvpx-vp9 -crf 30/libvpx -filter:v scale=720:-1,fps=10 -crf 10/' BatchConvertMp4.sh

3) convert

sh BatchConvertMp4.sh

--


The best possible quality that was still working in the "popsicle cutscene" for me was with the following two pass encoding VP9 in 1280x720 10fps 512kbps average
From the original file replace the line

ffmpeg -f lavfi -i anullsrc=r=8000:cl=mono -i "$SourceFile" -c:v libvpx-vp9 -crf 30 -b:v 0 -c:a libopus -b:a 8k -y -shortest "$TargetFile"

with:

if []; then
    ffmpeg  -f lavfi -i anullsrc=r=8000:cl=mono -i "$SourceFile" -c:v libvpx-vp9 -filter:v scale=1280:720,fps=10 -b:v 512K -pass 1 -passlogfile "$SourceFile"  -an -f null /dev/null
fi
ffmpeg -f lavfi -i anullsrc=r=8000:cl=mono -i "$SourceFile" -c:v libvpx-vp9 -filter:v scale=1280:720,fps=10 -b:v 512K -pass 2 -passlogfile "$SourceFile" -c:a libopus -b:a 8k -y -shortest "$TargetFile"

note that this will take a long time to encode.
I might be able to provide the log files to spare you the first encoding pass...

--

I am running the game with swapoff and have a terminal open with

earlyoom -m 10 --prefer '.*ES2-Win64-Shipp.*'

running in the background so that everspace takes the bullet first no matter what and the rest of the system stays up and running.
For reference please see: https://github.com/rfjakob/earlyoom

[Jarmer]

While I appreciate your effort on getting around this bug, I'm not sure transcoding the videos into 720 res / 10 fps / 0.5 meg bitrate is much of a help. That's potato quality. Wouldn't it just be better to skip them with F and then rewatch from the menu? That works fine for me and I don't really "miss" the videos as I'm rewatching them.