Freezing in every game, occasionally causing total system lockup

[DivideTheZero]

Hello, I've been having this problem ever since I got the Deck. Every game at some point, will lockup and I'll be forced to either quit the game, or occasionally force shut down the system. Possibly hardware issue?

The freezing occurs at seemingly random times, but *usually* within the first 15 minutes of launching a game. The game will freeze, the screen will go black for a sec, then when it comes back the game will be completely unresponsive. Audio still plays, and I can usually use the overlay and exit the game. Sometimes the overlay will also be unresponsive, and I have to force shut down the system. (Note: In these scenarios, I can still hear the overlay, I just can't see it)

This freeze occurs in *every single game I have tested*. H:ZD, Control, Hitman 3, No Man's Sky, Half Life 2, ME: Legendary, etc.

It appears like the GPU is crashing or resetting. Graphics settings and intensity appear to have zero impact on the issue. Happens just as often on low presets as on high presets.

So far I have tried re-imaging the deck, and opted into the beta branch. No change.

[deaddoof]

Originally posted by INIT_6_:

I've uploaded the coredumpctl logs to the same Github repo (https://github.com/INIT6Source/SteamDeckLogs).

OS Version: 3.5.13
OS Build: 20240104.1
Kernel Version: 6.1.52-valve14-1-neptune-61
BIOS Version: F7A0120

I've had the issue on both stable, beta, and preview update channels though. Only game played in the last few days was Pure Rock Crawling (https://store.steampowered.com/app/824720/Pure_Rock_Crawling/) but I was also having crashes on Sniper Elite 5 (https://store.steampowered.com/app/1029690/Sniper_Elite_5/) too.

Should I make both a Valve Steam OS bug report as well as opening an AMD bug issue? What relevant info/logs should I include with each?

I'll try and APU reset this evening and do some more Pure Rock Crawling to see if I can replicate the issue.

Thanks by the way! I figured the error was in there but I wasn't sure I was looking for specifically.

Report it on both bug trackers. Get both organizations attention and provide links to each other's bug trackers. Attach both journalctl logs to both bug reports. I blame AMD for putting so much power management logic in their closed firmware. Firmware has come and bite us back again. Community has no way to inspect the code quality of their firmware.

The lack of debug symbols annoy me

    Boot ID: bc1193086aa24c07af702133b265ec63
    Machine ID: 013519e787e24b8c886e4f3a7d510696
      Hostname: steamdeck
       Storage: /var/lib/systemd/coredump/core.gamescope-wl.1000.bc1193086aa24c07af702133b265ec63.1023.1704507114000000.zst (present)
  Size on Disk: 2.5M
       Message: Process 1023 (gamescope-wl) of user 1000 dumped core.
                
                Stack trace of thread 1088:
                #0  0x00007f294969f26c n/a (libc.so.6 + 0x8926c)
                #1  0x00007f294964fa08 raise (libc.so.6 + 0x39a08)
                #2  0x00007f2949638538 abort (libc.so.6 + 0x22538)
                #3  0x00007f294963845c n/a (libc.so.6 + 0x2245c)
                #4  0x00007f29496483d6 __assert_fail (libc.so.6 + 0x323d6)
                #5  0x000055ef32d86d97 n/a (gamescope + 0x7fd97)
                #6  0x000055ef32d61fe3 n/a (gamescope + 0x5afe3)
                #7  0x000055ef32d7cfac n/a (gamescope + 0x75fac)
                #8  0x00007f29498e1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                #9  0x00007f294969d44b n/a (libc.so.6 + 0x8744b)
                #10 0x00007f2949720e40 n/a (libc.so.6 + 0x10ae40)
                

I wonder whether you should combine these coredump logs to your jounralctl in the same bug or file a separate bug for GPU reset crashing game scope. I do not know. grey area.

[atmakeoutcreek]

Originally posted by INIT_6_:

Originally posted by deaddoof:

Can you dump out your logs?

How to find the logs?

[deaddoof]

Originally posted by atmakeoutcreek:

How to find the logs?

https://wiki.archlinux.org/title/Core_dump

https://wiki.archlinux.org/title/Systemd/Journal

journalctl can retrieve all the logs.

Coredumpctl records application crashes.

You can report bugs with these two commands and they are pretty useful for devs.

[INIT_6_]

Originally posted by deaddoof:

Originally posted by INIT_6_:

I've uploaded the coredumpctl logs to the same Github repo (https://github.com/INIT6Source/SteamDeckLogs).

OS Version: 3.5.13
OS Build: 20240104.1
Kernel Version: 6.1.52-valve14-1-neptune-61
BIOS Version: F7A0120

I've had the issue on both stable, beta, and preview update channels though. Only game played in the last few days was Pure Rock Crawling (https://store.steampowered.com/app/824720/Pure_Rock_Crawling/) but I was also having crashes on Sniper Elite 5 (https://store.steampowered.com/app/1029690/Sniper_Elite_5/) too.

Should I make both a Valve Steam OS bug report as well as opening an AMD bug issue? What relevant info/logs should I include with each?

I'll try and APU reset this evening and do some more Pure Rock Crawling to see if I can replicate the issue.

Thanks by the way! I figured the error was in there but I wasn't sure I was looking for specifically.

Report it on both bug trackers. Get both organizations attention and provide links to each other's bug trackers. Attach both journalctl logs to both bug reports. I blame AMD for putting so much power management logic in their closed firmware. Firmware has come and bite us back again. Community has no way to inspect the code quality of their firmware.

The lack of debug symbols annoy me

    Boot ID: bc1193086aa24c07af702133b265ec63
    Machine ID: 013519e787e24b8c886e4f3a7d510696
      Hostname: steamdeck
       Storage: /var/lib/systemd/coredump/core.gamescope-wl.1000.bc1193086aa24c07af702133b265ec63.1023.1704507114000000.zst (present)
  Size on Disk: 2.5M
       Message: Process 1023 (gamescope-wl) of user 1000 dumped core.
                
                Stack trace of thread 1088:
                #0  0x00007f294969f26c n/a (libc.so.6 + 0x8926c)
                #1  0x00007f294964fa08 raise (libc.so.6 + 0x39a08)
                #2  0x00007f2949638538 abort (libc.so.6 + 0x22538)
                #3  0x00007f294963845c n/a (libc.so.6 + 0x2245c)
                #4  0x00007f29496483d6 __assert_fail (libc.so.6 + 0x323d6)
                #5  0x000055ef32d86d97 n/a (gamescope + 0x7fd97)
                #6  0x000055ef32d61fe3 n/a (gamescope + 0x5afe3)
                #7  0x000055ef32d7cfac n/a (gamescope + 0x75fac)
                #8  0x00007f29498e1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                #9  0x00007f294969d44b n/a (libc.so.6 + 0x8744b)
                #10 0x00007f2949720e40 n/a (libc.so.6 + 0x10ae40)
                

I wonder whether you should combine these coredump logs to your jounralctl in the same bug or file a separate bug for GPU reset crashing game scope. I do not know. grey area.

Happened again this evening while playing Pure Rock Crawling - game froze up with sound still playing till the Deck itself crashed to black and rebooted. Grabbed the journalctl and coredump logs.

https://github.com/INIT6Source/SteamDeckLogs/blob/main/coredumpctl_1100_1_11_24

https://github.com/INIT6Source/SteamDeckLogs/blob/main/bootnum_1_11_24.txt

OS Version: 3.5.7
OS Build: 20231122.1
Kernel Version: 6.1.52-valve9-1-neptune-61
BIOS Version: F7A0120

[Rodomar705]

Originally posted by INIT_6_:

Originally posted by deaddoof:

Report it on both bug trackers. Get both organizations attention and provide links to each other's bug trackers. Attach both journalctl logs to both bug reports. I blame AMD for putting so much power management logic in their closed firmware. Firmware has come and bite us back again. Community has no way to inspect the code quality of their firmware.

The lack of debug symbols annoy me

    Boot ID: bc1193086aa24c07af702133b265ec63
    Machine ID: 013519e787e24b8c886e4f3a7d510696
      Hostname: steamdeck
       Storage: /var/lib/systemd/coredump/core.gamescope-wl.1000.bc1193086aa24c07af702133b265ec63.1023.1704507114000000.zst (present)
  Size on Disk: 2.5M
       Message: Process 1023 (gamescope-wl) of user 1000 dumped core.
                
                Stack trace of thread 1088:
                #0  0x00007f294969f26c n/a (libc.so.6 + 0x8926c)
                #1  0x00007f294964fa08 raise (libc.so.6 + 0x39a08)
                #2  0x00007f2949638538 abort (libc.so.6 + 0x22538)
                #3  0x00007f294963845c n/a (libc.so.6 + 0x2245c)
                #4  0x00007f29496483d6 __assert_fail (libc.so.6 + 0x323d6)
                #5  0x000055ef32d86d97 n/a (gamescope + 0x7fd97)
                #6  0x000055ef32d61fe3 n/a (gamescope + 0x5afe3)
                #7  0x000055ef32d7cfac n/a (gamescope + 0x75fac)
                #8  0x00007f29498e1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                #9  0x00007f294969d44b n/a (libc.so.6 + 0x8744b)
                #10 0x00007f2949720e40 n/a (libc.so.6 + 0x10ae40)
                

I wonder whether you should combine these coredump logs to your jounralctl in the same bug or file a separate bug for GPU reset crashing game scope. I do not know. grey area.

Happened again this evening while playing Pure Rock Crawling - game froze up with sound still playing till the Deck itself crashed to black and rebooted. Grabbed the journalctl and coredump logs.

https://github.com/INIT6Source/SteamDeckLogs/blob/main/coredumpctl_1100_1_11_24

https://github.com/INIT6Source/SteamDeckLogs/blob/main/bootnum_1_11_24.txt

OS Version: 3.5.7
OS Build: 20231122.1
Kernel Version: 6.1.52-valve9-1-neptune-61
BIOS Version: F7A0120

This is just a gfx ring crash. If it happens only on this specific game, it's either a game bug or a mesa bug.

[INIT_6_]

Originally posted by Rodomar705:

Originally posted by INIT_6_:

Happened again this evening while playing Pure Rock Crawling - game froze up with sound still playing till the Deck itself crashed to black and rebooted. Grabbed the journalctl and coredump logs.

https://github.com/INIT6Source/SteamDeckLogs/blob/main/coredumpctl_1100_1_11_24

https://github.com/INIT6Source/SteamDeckLogs/blob/main/bootnum_1_11_24.txt

OS Version: 3.5.7
OS Build: 20231122.1
Kernel Version: 6.1.52-valve9-1-neptune-61
BIOS Version: F7A0120

This is just a gfx ring crash. If it happens only on this specific game, it's either a game bug or a mesa bug.

Happens in several games. Steam Deck crashed a second time after this last night as well.

[Rodomar705]

Originally posted by INIT_6_:

Originally posted by Rodomar705:

This is just a gfx ring crash. If it happens only on this specific game, it's either a game bug or a mesa bug.

Happens in several games. Steam Deck crashed a second time after this last night as well.

If you can post the logs on another game crash I can take a look again if it's identical.

It's probably still the same power bug as reported above, but still, better double check :)

[INIT_6_]

Originally posted by Rodomar705:

Originally posted by INIT_6_:

Happens in several games. Steam Deck crashed a second time after this last night as well.

If you can post the logs on another game crash I can take a look again if it's identical.

It's probably still the same power bug as reported above, but still, better double check :)

Yup, I'll likely post all the logs of the various crashes I encounter. I don't know the logs well enough to read them or have the time but I'm happy to grab and upload them. Need to open bug reports still for the others.

[deaddoof]

Originally posted by INIT_6_:

Originally posted by Rodomar705:

If you can post the logs on another game crash I can take a look again if it's identical.

It's probably still the same power bug as reported above, but still, better double check :)

Yup, I'll likely post all the logs of the various crashes I encounter. I don't know the logs well enough to read them or have the time but I'm happy to grab and upload them. Need to open bug reports still for the others.

Its fine man. It took me years to be able to remotely read logs. Just file bug reports after the crash. Someone smarter than you may help. Your can learn from someone smarter. After awhile, you will notice a pattern.

[frankenchris]

This is what fixed it for me. I had to follow it to a T. Keep in mind even if you think you tried it already, a lot of people don’t realize they are not pressing in the … button completely, which causes it to not properly work.

In first place you are going to want to reset your APU's cache memory, and you can do that by powering down the steam deck completely, and once it is off you have to hold the Vol - button and the Quick Access button (...) while you power on the device.

KEEP HOLDING VOL - and (...) UNTIL YOU SEE THE STEAM LOGO.

When done correctly, you should hear about 1 to 2 chirps from the steam deck and it should take longer to boot. Keep in mind this process is going to take a bit longer than a regular boot, so I would say that if your steam deck is still at the logo after 20+ mins just go ahead and repeat the process. (I had this issue once where it got stuck at the steam logo screen without rebooting, if that happens just power it down and repeat the process). This process should reset your APU's cache memory.

[vittau]

I had a crash like this over a month ago, it happened on the SteamOS download page, so I wasn't even gaming.

I used it a lot in between, and today it happened again playing a very lightweight game (Turnip Boy Robs a Bank).

Should I be worried? I own this Deck since July, haven't had any other issues. Is it normal to get these crashes once in a while without it being indicative of hardware failure?

I'm in Brazil so there's no way to RMA it.

[deaddoof]

Originally posted by vittau:

I had a crash like this over a month ago, it happened on the SteamOS download page, so I wasn't even gaming.

I used it a lot in between, and today it happened again playing a very lightweight game (Turnip Boy Robs a Bank).

Should I be worried? I own this Deck since July, haven't had any other issues. Is it normal to get these crashes once in a while without it being indicative of hardware failure?

I'm in Brazil so there's no way to RMA it.

Crash or system lockup?

[vittau]

Originally posted by deaddoof:

Originally posted by vittau:

I had a crash like this over a month ago, it happened on the SteamOS download page, so I wasn't even gaming.

I used it a lot in between, and today it happened again playing a very lightweight game (Turnip Boy Robs a Bank).

Should I be worried? I own this Deck since July, haven't had any other issues. Is it normal to get these crashes once in a while without it being indicative of hardware failure?

I'm in Brazil so there's no way to RMA it.

Crash or system lockup?

Same as usually reported here, pretty much.
Freezes for a couple seconds, comes back, works for a couple seconds, freezes again to black and stays. I think sound was working while it was frozen (on black screen it wasn't).
First case it rebooted itself, second case Ij forced reset with the power button.
In both cases I got the "verifying installation" screen afterwards, with no issues found I believe.

I believe these are GPU freezes too, just not sure if "benign" driver-related bugs or whatever, or hardware-related.

EDIT: I have Decky installed, only with SteamGridDB. CryoUtilities settings all enabled.
Still haven't tried any sort of troubleshooting like the APU cache reset combination. The issue only happened twice with 36 days in between, so too sporadic to debug.

[deaddoof]

Originally posted by vittau:

Originally posted by deaddoof:

Crash or system lockup?

Same as usually reported here, pretty much.
Freezes for a couple seconds, comes back, works for a couple seconds, freezes again to black and stays. I think sound was working while it was frozen (on black screen it wasn't).
First case it rebooted itself, second case Ij forced reset with the power button.
In both cases I got the "verifying installation" screen afterwards, with no issues found I believe.

I believe these are GPU freezes too, just not sure if "benign" driver-related bugs or whatever, or hardware-related.

EDIT: I have Decky installed, only with SteamGridDB. CryoUtilities settings all enabled.
Still haven't tried any sort of troubleshooting like the APU cache reset combination. The issue only happened twice with 36 days in between, so too sporadic to debug.

Sounds like a gpu reset. You kinda need to grab logs from journalctl. Damn, it sounds like you have a rainbow of problems. I would recommend attempt the APU reset. It will reset so many bios settings.

https://wiki.archlinux.org/title/Systemd/Journal#Filtering_output

I know one widespread issue but its a complete system lock up here. This problem is extremely technical to even report it at all.

https://gitlab.freedesktop.org/drm/amd/-/issues/3111

Yea. I would contact Valve. Please grab your journalctl logs.

https://github.com/ValveSoftware/SteamOS/issues

[vittau]

Originally posted by deaddoof:

Originally posted by vittau:

Same as usually reported here, pretty much.
Freezes for a couple seconds, comes back, works for a couple seconds, freezes again to black and stays. I think sound was working while it was frozen (on black screen it wasn't).
First case it rebooted itself, second case Ij forced reset with the power button.
In both cases I got the "verifying installation" screen afterwards, with no issues found I believe.

I believe these are GPU freezes too, just not sure if "benign" driver-related bugs or whatever, or hardware-related.

EDIT: I have Decky installed, only with SteamGridDB. CryoUtilities settings all enabled.
Still haven't tried any sort of troubleshooting like the APU cache reset combination. The issue only happened twice with 36 days in between, so too sporadic to debug.

Sounds like a gpu reset. You kinda need to grab logs from journalctl. Damn, it sounds like you have a rainbow of problems. I would recommend attempt the APU reset. It will reset so many bios settings.

https://wiki.archlinux.org/title/Systemd/Journal#Filtering_output

I know one widespread issue but its a complete system lock up here. This problem is extremely technical to even report it at all.

https://gitlab.freedesktop.org/drm/amd/-/issues/3111

Yea. I would contact Valve. Please grab your journalctl logs.

https://github.com/ValveSoftware/SteamOS/issues

Found a
ring gfx_0.0.0 timeout, but soft recovered

So that confirms a GPU freeze, but that's sort of expected. I don't have the "ring sdma0 timeout" cases like issue you linked though.