Steam overlay not working on GeForce NOW

[Jim29er]

As of this writing, the Steam overlay is not functional when playing Terraria on NVIDIA’s GeForce NOW (GFN) cloud gaming service. The specific details are in this community forum post, Terraria Is Extremely Broken [www.nvidia.com]. I would really like to get to the bottom of the “why” this is happening, so that it can potentially be fixed.

The TL;DR is that one “root cause” of this issue, is related to the Windows “application compatibility” functionality in the OS…when the game is launched for the first time. I believe this to be the reason the issue persists on the GFN service, because gaming sessions there use ephemeral virtual machines. Every gaming session is essentially being run on a new computer. What I don’t know, is how to stop it from happening on the “first” launch. The information below details how I came to this conclusion.

There have been many posts (and various “solutions”) over the years, about the Steam overlay not working in Terraria for people using local computers. While I’m trying to find a way the issue can be prevented on the GFN service, there is some overlap with the issue on local systems. As such, I think the information I’m writing here can be helpful in chasing it down on those systems as well.

While troubleshooting the Steam overlay issue with this game, I noticed a pattern. After installing the game through Steam, on a local computer which has never had the game installed previously…the Steam overlay did not work during the first launch of the game. But on subsequent launches, the Steam overlay worked fine. I reproduced this on three different Windows computers.

After seeing this pattern, I provisioned a new Windows 11 22H2 virtual machine inside of Hyper-V on my laptop. I installed Steam, Terraria, and the Microsoft XNA Framework Redistributable 4.0. The latter is included with the game, and I installed it manually so that Steam wouldn’t prompt to install it when running Terraria for the first time. Then, I took a checkpoint of the VM (Hyper-V’s term for a snapshot), so that I could keep going back to this state. Now, I can run Terraria in the virtual machine, and the Steam overlay does not work during the first launch of the game, but does work on subsequent launches. When I revert to the checkpoint, the same thing happens. The Steam overlay does not work on the first launch of the game, ever.

Since the Steam overlay DOES work on the subsequent launches of the game, we know that “something” is being changed after the first launch…but that “something” has been elusive to me. I have tried using a “whatchanged” utility, to capture changes to files/registry in the VM after Terraria has been launched once. I believe I’ve chased them all down, but undoing those changes individually does not result in the Steam overlay being broken again. As for the reverse…applying those changes to the computer prior to the first launch (after restoring the checkpoint)…that is still on the to-do list. If I happen upon a solution there, I will come back and update this thread.

Anyway, I thought it would be valuable to call out some items I’ve seen after going through this process. The most significant, is the “C:\Program Files (x86)\Steam\GameOverlayRenderer.log” file. When Terraria is launched for the first time, this log file looks like this:

Wed Mar 22 05:33:41 2023 UTC - Module file name: C:\Program Files (x86)\Steam\gameoverlayrenderer.dll
Wed Mar 22 05:33:41 2023 UTC - GameID = 105600, OverlayGameID = 105600
Wed Mar 22 05:33:41 2023 UTC - System page size: 4096
Wed Mar 22 05:33:41 2023 UTC - Failed loading shell32.dll, not hooking ShellExecute calls
Wed Mar 22 05:33:41 2023 UTC - Hooking SetCursorPos, GetCursorPos, ShowCursor, SetCursor, and GetCursor
Wed Mar 22 05:33:41 2023 UTC - Modules at GameOverlayRenderer.dll attach
Wed Mar 22 05:33:41 2023 UTC - 01: Terraria.exe - (00620000 to 019A0000)
Wed Mar 22 05:33:41 2023 UTC - 02: ntdll.dll - (777F0000 to 7799F000)
Wed Mar 22 05:33:41 2023 UTC - 03: MSCOREE.DLL - (6D4E0000 to 6D535000)
Wed Mar 22 05:33:41 2023 UTC - 04: KERNEL32.dll - (76150000 to 76240000)
Wed Mar 22 05:33:41 2023 UTC - 05: KERNELBASE.dll - (770D0000 to 7733A000)
Wed Mar 22 05:33:41 2023 UTC - 06: apphelp.dll - (6D540000 to 6D5E5000)
Wed Mar 22 05:33:41 2023 UTC - 07: gameoverlayrenderer.dll - (6D390000 to 6D4D9000)
Wed Mar 22 05:33:41 2023 UTC - 08: USER32.dll - (75580000 to 75727000)
Wed Mar 22 05:33:41 2023 UTC - 09: win32u.dll - (75850000 to 7586A000)
Wed Mar 22 05:33:41 2023 UTC - 10: GDI32.dll - (76360000 to 76382000)
Wed Mar 22 05:33:41 2023 UTC - 11: gdi32full.dll - (76390000 to 7646E000)
Wed Mar 22 05:33:41 2023 UTC - 12: msvcp_win.dll - (75D40000 to 75DB9000)
Wed Mar 22 05:33:41 2023 UTC - 13: ucrtbase.dll - (75FD0000 to 760E2000)
Wed Mar 22 05:33:41 2023 UTC - 14: ADVAPI32.dll - (76510000 to 7658C000)
Wed Mar 22 05:33:41 2023 UTC - 15: msvcrt.dll - (76F70000 to 77034000)
Wed Mar 22 05:33:41 2023 UTC - 16: sechost.dll - (77040000 to 770C2000)
Wed Mar 22 05:33:41 2023 UTC - 17: RPCRT4.dll - (77700000 to 777B9000)
Wed Mar 22 05:33:41 2023 UTC - 18: ole32.dll - (76590000 to 766DD000)
Wed Mar 22 05:33:41 2023 UTC - 19: combase.dll - (77480000 to 776FB000)
Wed Mar 22 05:33:41 2023 UTC - 20: OLEAUT32.dll - (75F30000 to 75FCC000)
Wed Mar 22 05:33:41 2023 UTC - 21: IMM32.dll - (75D10000 to 75D35000)
Wed Mar 22 05:33:41 2023 UTC - 22: PSAPI.DLL - (777C0000 to 777C6000)
Wed Mar 22 05:33:41 2023 UTC - 23: WINMM.dll - (738D0000 to 73901000)
Wed Mar 22 05:33:41 2023 UTC - ----------------------------
Wed Mar 22 05:33:42 2023 UTC - GetProcAddress Direct3DCreate9 returned wrong module (6D540000='C:\Windows\SYSTEM32\apphelp.dll'), proceeding anyway
Wed Mar 22 05:33:42 2023 UTC - Game is using D3D9 or D3D9Ex, preparing to hook.
Wed Mar 22 05:33:42 2023 UTC - GetProcAddress Direct3DCreate9Ex returned wrong module (6D540000='C:\Windows\SYSTEM32\apphelp.dll'), proceeding anyway
Wed Mar 22 05:33:45 2023 UTC - Initializing Audio...
Wed Mar 22 05:34:15 2023 UTC - XInput Hooked XInputEnable Version 13
Wed Mar 22 05:34:15 2023 UTC - XInput Hooked XInputGetBatteryInformation Version 13
Wed Mar 22 05:34:15 2023 UTC - XInput Hooked XInputGetCapabilities Version 13
Wed Mar 22 05:34:15 2023 UTC - XInput Hooked XInputGetDSoundAudioDeviceGuids Version 13
Wed Mar 22 05:34:15 2023 UTC - XInput Hooked XInputGetKeystroke Version 13
Wed Mar 22 05:34:15 2023 UTC - XInput Hooked XInputGetState Version 13
Wed Mar 22 05:34:15 2023 UTC - XInput Hooked XInputGetStateEX Version 13
Wed Mar 22 05:34:15 2023 UTC - XInput Hooked XInputSetState Version 13
Wed Mar 22 05:34:25 2023 UTC - Injecting to a child process...
Wed Mar 22 05:34:26 2023 UTC - GameOverlayRenderer.dll detaching 

Then, when Terraria is launched for the second time, the log file looks like this:

Wed Mar 22 05:36:03 2023 UTC - Module file name: C:\Program Files (x86)\Steam\gameoverlayrenderer.dll
Wed Mar 22 05:36:03 2023 UTC - GameID = 105600, OverlayGameID = 105600
Wed Mar 22 05:36:03 2023 UTC - System page size: 4096
Wed Mar 22 05:36:03 2023 UTC - Failed loading shell32.dll, not hooking ShellExecute calls
Wed Mar 22 05:36:03 2023 UTC - Hooking SetCursorPos, GetCursorPos, ShowCursor, SetCursor, and GetCursor
Wed Mar 22 05:36:03 2023 UTC - Modules at GameOverlayRenderer.dll attach
Wed Mar 22 05:36:03 2023 UTC - 01: Terraria.exe - (00550000 to 018D0000)
Wed Mar 22 05:36:03 2023 UTC - 02: ntdll.dll - (777F0000 to 7799F000)
Wed Mar 22 05:36:03 2023 UTC - 03: MSCOREE.DLL - (6D4E0000 to 6D535000)
Wed Mar 22 05:36:03 2023 UTC - 04: KERNEL32.dll - (76150000 to 76240000)
Wed Mar 22 05:36:03 2023 UTC - 05: KERNELBASE.dll - (770D0000 to 7733A000)
Wed Mar 22 05:33:41 2023 UTC - 06: gameoverlayrenderer.dll - (6D390000 to 6D4D9000)
Wed Mar 22 05:36:03 2023 UTC - 07: USER32.dll - (75580000 to 75727000)
Wed Mar 22 05:36:03 2023 UTC - 08: win32u.dll - (75850000 to 7586A000)
Wed Mar 22 05:36:03 2023 UTC - 09: GDI32.dll - (76360000 to 76382000)
Wed Mar 22 05:36:03 2023 UTC - 10: gdi32full.dll - (76390000 to 7646E000)
Wed Mar 22 05:36:03 2023 UTC - 11: msvcp_win.dll - (75D40000 to 75DB9000)
Wed Mar 22 05:36:03 2023 UTC - 12: ucrtbase.dll - (75FD0000 to 760E2000)
Wed Mar 22 05:36:03 2023 UTC - 13: ADVAPI32.dll - (76510000 to 7658C000)
Wed Mar 22 05:36:03 2023 UTC - 14: msvcrt.dll - (76F70000 to 77034000)
Wed Mar 22 05:36:03 2023 UTC - 15: sechost.dll - (77040000 to 770C2000)
Wed Mar 22 05:36:03 2023 UTC - 16: RPCRT4.dll - (77700000 to 777B9000)
Wed Mar 22 05:36:03 2023 UTC - 17: ole32.dll - (76590000 to 766DD000)
Wed Mar 22 05:36:03 2023 UTC - 18: combase.dll - (77480000 to 776FB000)
Wed Mar 22 05:36:03 2023 UTC - 19: OLEAUT32.dll - (75F30000 to 75FCC000)
Wed Mar 22 05:36:03 2023 UTC - 20: IMM32.dll - (75D10000 to 75D35000)
Wed Mar 22 05:36:03 2023 UTC - 21: PSAPI.DLL - (777C0000 to 777C6000)
Wed Mar 22 05:36:03 2023 UTC - 22: WINMM.dll - (738D0000 to 73901000)
Wed Mar 22 05:36:03 2023 UTC - ----------------------------
Wed Mar 22 05:36:04 2023 UTC - Game is using D3D9 or D3D9Ex, preparing to hook.
Wed Mar 22 05:36:05 2023 UTC - hookDirect3DCreate9 called
Wed Mar 22 05:36:05 2023 UTC - IWrapIDirect3D9::CreateDevice hook called (type: 1, behavior flags: 46!)
Wed Mar 22 05:36:05 2023 UTC - Creating D3D9 renderer
Wed Mar 22 05:36:05 2023 UTC - Releasing all resources for device: 0D7CF680
Wed Mar 22 05:36:05 2023 UTC - Initializing Audio...
Wed Mar 22 05:36:06 2023 UTC - Trying to setup input hook...
Wed Mar 22 05:36:06 2023 UTC - Set input hook...
Wed Mar 22 05:36:23 2023 UTC - XInput Hooked XInputEnable Version 13
Wed Mar 22 05:36:23 2023 UTC - XInput Hooked XInputGetBatteryInformation Version 13
Wed Mar 22 05:36:23 2023 UTC - XInput Hooked XInputGetCapabilities Version 13
Wed Mar 22 05:36:23 2023 UTC - XInput Hooked XInputGetDSoundAudioDeviceGuids Version 13
Wed Mar 22 05:36:23 2023 UTC - XInput Hooked XInputGetKeystroke Version 13
Wed Mar 22 05:36:23 2023 UTC - XInput Hooked XInputGetState Version 13
Wed Mar 22 05:36:23 2023 UTC - XInput Hooked XInputGetStateEX Version 13
Wed Mar 22 05:36:23 2023 UTC - XInput Hooked XInputSetState Version 13
Wed Mar 22 05:36:26 2023 UTC - Detected hot-key via base input, now requesting overlay enable
Wed Mar 22 05:36:27 2023 UTC - Showing overlay and saving cursor show count: -1
Wed Mar 22 05:36:27 2023 UTC - Saved Cursor: 00010003
Wed Mar 22 05:36:27 2023 UTC - Saved GCLP_HCURSOR: 0
Wed Mar 22 05:36:27 2023 UTC - Paused device notifications
Wed Mar 22 05:36:29 2023 UTC - Hiding overlay and restoring cursor show count: -1
Wed Mar 22 05:36:29 2023 UTC - Restoring Saved Cursor: 00010003
Wed Mar 22 05:36:29 2023 UTC - Restoring clip cursor region: -32768,-32768 32767,32767
Wed Mar 22 05:36:29 2023 UTC - Performing 0 queued raw input registrations
Wed Mar 22 05:36:29 2023 UTC - Resumed device notifications
Wed Mar 22 05:36:31 2023 UTC - Releasing all resources for device: 0D7CF680
Wed Mar 22 05:36:31 2023 UTC - Detaching input hook...
Wed Mar 22 05:36:32 2023 UTC - Injecting to a child process...
Wed Mar 22 05:36:32 2023 UTC - GameOverlayRenderer.dll detaching

The suspicious entries in the first log file, have to do with the 'C:\Windows\SYSTEM32\apphelp.dll' lines. This DLL file is the Windows “Application Compatibility Client Library.” It would seem, that this thing is interfering with Steam’s GameOverlayRenderer.dll hooking into the game properly, during the first launch. So, how to “prove” this?

Here is what I did. There are two copies of that DLL file:

C:\Windows\SYSTEM32\apphelp.dll
C:\Windows\SysWOW64\apphelp.dll

In the virtual machine, I took ownership of these DLL files in the advanced security properties, and then changed the permissions on them removing all objects…basically making it so these DLL files become unusable by any users/Windows itself. Then, when Terraria is launched for the first time…the Steam overlay works! However…this is not a “solution,” as we can’t just hobble a Windows component like this. I did look at other methods to try and disable this Windows app compatibility thing, such as disabling the “Program Compatibility Assistant” service prior to the game launch (didn’t work), and using the Application Compatibility Toolkit in the Windows ADK to look for any entries for the (32-bit) Terraria processes…which also turned up nothing.

I also had a look at other games being launched for the first time, and for one in particular (LIMBO) there are apphelp.dll entries in the GameOverlayRenderer.log, but in the “upper” not lower section…and the Steam overlay DOES work on the first launch. It seems this “first launch overlay” issue is isolated to Terraria.

In conclusion, we have circumstantial evidence that the Windows “application compatibility” feature is what breaks the Steam overlay during the first launch of Terraria…but don’t know how to prevent it from happening in the first place. I believe this is the “holy grail” I’m looking for.

Also, whose issue is this to address? Microsoft/Valve/Re-logic ?

If anyone has any ideas or suggestions for chasing this down, please let me know!

Thanks for reading!