Update linux dedicated server Segmentation Fault

[Kanopy]

I've been running a dedicated linux server for a number of months. The update today came out and my users couldn't connect as their clients were updated. So i updated the server using the installupdate.sh script as per https://valheim.fandom.com/wiki/Valheim_Dedicated_Server here, but just after the server starts it throws a segmentation fault:

#45 0x007f5a284f2ebe in PlayerLoop()
#46 0x007f5a286d0bab in PlayerMain(int, char**)
#47 0x007f5a26c0fc87 in __libc_start_main
#48 0x0055c8c63e5029 in _start
./startval.sh: line 9: 16967 Segmentation fault (core dumped) /home/valheim/valheim-new/valheim_server.x86_64 -name "Valheim Brass" -port 2456 -nographics -batchmode -world "ValheimBrass" -password "*****" -public 1

The installation of the server was done using SteamCMD using following command:

#!/bin/sh
steamcmd +@sSteamCmdForcePlatformType linux +force_install_dir /path/to/server +login anonymous +app_update 896660 validate +quit

Anyone got any clues or has Valheim team broken their linux build?

[UPDATE] - after much faffing around, the issue is all to do with one single library "libparty.so". This library is not developed by IronGate but by Microsoft in order to use their FabParty, this doesn't appear to be very stable at all on any of the linux distros (i've tried all versions of ubuntu and debian) and there are problems with each. Earlier version just throw a segmentation fault, later versions it runs, but you get disconnected from the server a number of times. By far, the easiest fix I've found so far is just to remove the library completely and remove the -crossplay option from the start script that they add.

rm <valheim home directory>/valheim_server_Data/Plugins/libparty.so
edit start_server.sh and remove '-crossplay' option from start line.

Final comment: it would have been extremely nice, if the developers could have provided some information on the platforms this library has been tested on - because it clearly wasn't ubuntu or debian: two of the most popular linux distros there is!

[Kanopy]

I should probably also add this is running on Ubuntu Linux VERSION="18.04.6 LTS (Bionic Beaver)"

[Dr Nick]

Same issue here with Ubuntu. If we can no longer use Ubuntu, what version(s) of Linux do you support?

Thanks

[astroom]

If this is true, removing support for Ubuntu is absolutely ludacris. 90% of all game hosts use that as their base OS. Massive mistake and I really hope this decision is reverted as soon as possible

[not_amused]

Glad I found this information here. Subscribing hoping that a solution for Ubuntu servers is identified. Lots of us.

[not_amused]

According to patch notes it seems playfab is opt-in with the ‘-crossplay’ parameter? Is that not the case?

>>> * Players can now use the parameter ‘-crossplay’ while running Dedicated servers to support crossplay. When using this parameter, the backend will be running Playfab instead of Steamworks.

[t0mpre]

I've checked this update on the newest Ubuntu LTE 22.04 -
Fresh Steam headless installation + valheim server up n running.

[Dinkleberg]

Thanks for making this thread. Having the same issue, on Debian Stretch. Upgrading to Buster now to see if it works.

Would have loved for this requirement to be published, it's still not clear which distros are supported.

[Kanopy]

Originally posted by t0mpre:

I've checked this update on the newest Ubuntu LTE 22.04 -
Fresh Steam headless installation + valheim server up n running.

Thanks - are you able to provide the install script and start script you used? I've tried with Ubuntu 20.04 now on a newer VM, it doesn't core dump any more, but it halts after these lines when starting up:

Loading in SingleInstance mode
Mono path[0] = '/home/valheim/valheim/valheim_server_Data/Managed'
Mono config path = '/home/valheim/valheim/valheim_server_Data/MonoBleedingEdge/etc'
Preloaded 'libsteam_api.so'
Unable to preload the following plugins:
libparty.so
Initialize engine version: 2020.3.33f1 (915a7af8b0d5)
[Subsystems] Discovering subsystems at path /home/valheim/valheim/valheim_server_Data/UnitySubsystems
Forcing GfxDevice: Null
GfxDevice: creating device client; threaded=0
NullGfxDevice:
Version: NULL 1.0 [1.0]
Renderer: Null Device
Vendor: Unity Technologies
Begin MonoManager ReloadAssembly
- Completed reload, in 0.132 seconds
ERROR: Shader Sprites/Default shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader Sprites/Mask shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
ERROR: Shader GUI/Text Shader shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
UnloadTime: 1.646546 ms
09/27/2022 21:55:51: Starting to load scene:start
ERROR: Shader UI/Default shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
Couldn't create a Convex Mesh from source mesh "default" within the maximum polygons limit (256). The partial hull will be used. Consider simplifying your mesh.
Couldn't create a Convex Mesh from source mesh "default" within the maximum polygons limit (256). The partial hull will be used. Consider simplifying your mesh.

[Dr Nick]

Originally posted by t0mpre:

I've checked this update on the newest Ubuntu LTE 22.04 -
Fresh Steam headless installation + valheim server up n running.

Cheers, just upgraded to 20.04.5 LTS and have the server running now and a client connected. However, I have lost the world. :(

It's late and that's a tomorrow problem now. :)

[Kanopy]

Originally posted by Kanopy:

Originally posted by t0mpre:

I've checked this update on the newest Ubuntu LTE 22.04 -
Fresh Steam headless installation + valheim server up n running.

Thanks - are you able to provide the install script and start script you used? I've tried with Ubuntu 20.04 now on a newer VM, it doesn't core dump any more, but it halts after these lines when starting up:

Couldn't create a Convex Mesh from source mesh "default" within the maximum polygons limit (256). The partial hull will be used. Consider simplifying your mesh.
Couldn't create a Convex Mesh from source mesh "default" within the maximum polygons limit (256). The partial hull will be used. Consider simplifying your mesh.

Actually it does load eventually but it takes around 5 to 10 minutes before it's ready to be logged into!!

So to confirm it looks like Ubuntu LTE 18.04 is not supported, but 20.04 and upwards is - hope this helps everyone.

[-=[FR]BERT-7=-]

Damn they really screwed up.... Same issue here... I've a server running on Ubuntu with a tone of different apps... zero warning from them... this is extremely deceiving and unprofessional on their part...

Thank you for the post !

[Pincerbottom]

Originally posted by not_amused:

According to patch notes it seems playfab is opt-in with the ‘-crossplay’ parameter? Is that not the case?

>>> * Players can now use the parameter ‘-crossplay’ while running Dedicated servers to support crossplay. When using this parameter, the backend will be running Playfab instead of Steamworks.

When I did the install on a dedicated server, crossplay was enabled by default and the .sh script was altered to parameters that changed the world name, password, crossplay status, and public status.

Check the base startup.sh command and see if it was over-written. If you're using a .bat file that references that file, it may have parameters that are starting playfab by default. You might not want to do that if you have people logging in using an IP address/etc, or if you're using a hosting company (like, say, Bisect).

It would have been REALLY helpful for the Devs to specify what the update was going to do by default to dedicated servers.

[Dinkleberg]

Originally posted by Dinkleberg:

Thanks for making this thread. Having the same issue, on Debian Stretch. Upgrading to Buster now to see if it works.

Would have loved for this requirement to be published, it's still not clear which distros are supported.

Updated to Debian 10 LTS (buster), issue didn't go away Might try to do 11 eventually, but for now just downgraded the valheim server and told my friends to opt out of the client update.

[He Who Has No Name]

I'm running an 18.04 LTS server that has a lot more jobs than just Valheim, I'm not really sold on going through the process of doing an entire OS update just to prevent crashes by a feature that I'm not even going to use. I'm not even sure the server can do 20.04 gracefully, it's older hardware. Runs the dedicated server just fine, though... or did.

[thecsapprentice]

I think I'm getting similar issues on my local client too. Since the update, I've been unable to launch on Debian Buster. The issue seems to be connected to PlayFab, specifically the libparty.so. When I attempt to start the game, I get a message in the Player.log stating that it failed to preload the shared library and shortly later crashes with a very short call stack of 2 items.

Checking the libparty.so shipped with the new update with ldd, it looks to be fairly clear what the issue is. The shared library is looking for a newer version of GLIBC/GLIBCXX than is available on my system.

ldd libparty.so 
./libparty.so: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by ./libparty.so)
./libparty.so: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./libparty.so)

That's probably why their builds aren't "broken" - because they are using a newer linux install with more current system libraries. Further, investigating PlayFab leads me to a github repo from Microsoft where the libparty.so is probably coming from as a binary blob. If so, then it's not even the dev's doing this directly, but whatever build platform the Microsoft releases are coming from.

https://github.com/PlayFab/PlayFabParty

This is frustrating on two levels:

• First, the update was specified as "small" but failed to account for potentially new restrictions on platform requirements due to a new third party library (which seemingly has only had two releases for linux so far, the first being end of June). New third party code always comes with dependency risk, and this was clearly not anticipated fully.
  
• Second, the library compatibility is different from the GLIBC required by the steam platform. Unless there's a pressing need for a new GLIBC version (which seems unlikely unless there's a critical bug fix they depend on), linux builds should be compatible with the versions standardized by steam: https://partner.steamgames.com/doc/store/application/platforms/linux

Failure to do this breaks user expectations of what a game needs to run. If steam runs on a particular linux distro instance, any native games for linux should also be capable of running, at least in terms of needing the same version of system libraries for any of their components. If this absolutely can't happen, either due to incompatiblities or lack of development resources, that's fine, but it should be called out in updates that a new minimum version or recommended major distro version has been changed. A minimum requirement of "Any up to date OS" doesn't really differentiate between latest distro and older, but still receiving security and bug fix update distros. Version changes to things like GLIBC and GLIBCXX usually don't happen on the latter, for exactly this compatibility reason.

Edit - alternatively, if the cross play can be made optional, in that if the required library is missing or doesn't load properly, to continue and fall back to older approaches, that would be very nice as well. Since it's being loaded as a plugin, it should be possible to isolate it to some degree internally and build a failure tolerant abstraction layer.

At this point I'm going to simply wait on this. I love this game, but I'm not going to go through the annoyance of updating my distro at this level just to play it due to an update. Which is a shame, because it's long been a solid linux native game for me. Hopefully this information helps someone else!