Dedicated Server Crashing

[Zaish]

I am hoping someone can help. Sorry if this is long, I have been trying different things along with joining all the discord channels and reading outstanding issues.

I host my own VMWare Windows 10 PvE 12 map cluster for 5 people using ASM. One of the instances in the cluster, Genesis 1, crashes when I get near my base in the Ocean Biomes. If I spawn directly to my bed, I spawn on the grass, my health is low, my base is gone, and all other mod attached items are gone. The instance on the server crashes, along with RCON being disconnected and kicking everyone off the server. Users who were on this map are only given a timeout error. Genesis 1 still shows as online on the client side and that users are still connected. However, no one is able to re-connect. On the server side, the console window doesn't show that anyone disconnected and no other messages appear. This state lasts for around 15 - 30 minutes, I am than presented with this vague popup error message:

Fatal error!
VERSION: 358.17
ShooterGameServer.exe!FHangThreadWatcher::HangeDetected() (0x00007ff6e5ffa1f0) + 0 bytes [f:\build\lostisland\engine\source\runtime\engine\private\gameengine.cpp:172]
ShooterGameServer.exe!FHangThreadWatcher::Run() (0x00007ff6e5ffa143) + 0 bytes [f:\build\lostisland\engine\source\runtime\engine\private\gameengine.cpp:136]
ShooterGameServer.exe!FRunnableThreadWin::Run() (0x00007ff6e5a31caf) + 0 bytes [f:\build\lostisland\engine\source\runtime\core\private\windows\windowsrunnablethread.cpp:73]
ShooterGameServer.exe!FRunnableThreadWin::GuardedRun() (0x00007ff6e5a31ba8) + 8 bytes [f:\build\lostisland\engine\source\runtime\core\private\windows\windowsrunnablethread.cpp:26]
KERNEL32.DLL!UnknownFunction (0x00007ffc77777344) + 0 bytes [UnknownFile:0]
ntdll.dll!UnknownFunction (0x00007ffc78ae26b1) + 0 bytes [UnknownFile:0]
ntdll.dll!UnknownFunction (0x00007ffc78ae26b1) + 0 bytes [UnknownFile:0]
慌瑳删灥楬慣整⁤捁潴⁲畂晦䍟浯慰楮湯䡟乌彁敇ㅮ䍟た਍慌瑳删浥瑯⁥畆据楴湯䌠楬湥噴牥卹潨瑲摁番瑳潐楳楴湯਍

I use the following Mods in the below boot order:

1565015734 - Krackens Better Dino
1522327484 - ARK Additions: The Collection
1445395055 - ARK Additions: Domination Rex
2212177129 - Sid's Hybrids
1609138312 - Dino Storage V2
1931415003 - Best Egg
1967741708 - Lethals Reusable
2044129379 - Tribut Transfer
2802804366 - Better Tek Abilities
731604991 - S+
1136125765 - Bitou2k's Binocular
2859999627 - Player Finder
2769908652 - Better Waypoints V2

My base is built on land and is it pretty small. I use the S+ propegator, dino storage, best egg to breed. s+ Vivarium, Item collector, and S+ dedicated storage to generate and hold resources. Along with a land base, I also have a few ocean platforms to breed my water dinos. Across Genesis 1, I also have small teleporter bases in the differant biomes.

Whats odd is if I spawn at one of the other locations or even my ocean platform below... everything is as it should be. The minute I attempt to visit my main base, portions dont render making me think a bomb went off. It seems to be RNG which foundation pieces render.

Troubleshooting:

1. Verified all steam and OS files. The same issue still existed.
2. Performed a 2 day memory stress test on the server that runs this. Nothing found.
3. Created a new instance in ASM, re-downloaded a new copy outside of the cache, and copied save over. The same issue still existed.
4. Copied the save and ini settings to another computer not running on the server. I could reproduce the same issue. This lead me to believe this isnt hardware or OS related.
5. Adjusted the following in ASM: Changed 1x Dino Spawn setting under environments to 0x. Enabled No Dino spawn. Issued DestroyWildDinos via rcon, saving, restarting. Enabled No hand Detection. The same issue occurred.. I tested every 2 hour save for the last 12 hours to see if this started at a certain point. The same issue still existed.
6. Using the last backup over and over, I tested each scenario below:
a) Removed all the mods except for Dino Storage and S+. The same issue still existed.
b) I removed S+ but left Dino Storage. I no longer crashed and I was able to access all my dinos stored in the soul terminals floating in the air.
c) Added all the mods back, except for S+. The server no longer crashed and I was able to access all the other stations used by the mods listed above. Each item was floating in the air. Modded Dinos in the base reappeared.
d) Switched from S+ to SS. Everything loaded except for my dedicated storage,Vivarium,and Propagator. Modded Dino in the base reappeared. The server no longer crashed.
e) Adjusted boot order of the S+ mod to the top. Original issue returned.


Question:

1. Does anyone know of any additional debugging I could do? Based on my tests, it may be related to S+. However, I am unable to see if anyone else has the same issue as myself since the discussion board has been shutdown and no discord server is available. Ideally I could narrow it down a bit more. I also have been unable to get something better than what I presented above.

2. Can I use different mods on clustered servers? I would think that running identical mods across the cluster would be ideal. I am not sure if this could lead to corrupted save when you transfer your character over.


Additional information: I had the same issue on another map. We have recently just started using a few of the advance S+ items in our base. In the end I had to leave extinction and recreate the map. Not much is stored there now, compared to what we previously setup. https://www.reddit.com/r/ARK/comments/18k8hoo/ase_modded_server_crashing/.

If you made it this far, thank you for your help.

p.s. If you have never seen a fish fly out of the water while you are waiting for that timeout... Your missing out on the little things.

[Scramble]

Your D does make it seem to be one of the S+ items. Which Dedicated Storage do you use?

S+ have both the Tek and the older one. since there has been a fundamental change in how the storage works (server side they dropped having 2000 stacks and made it just 1 stack to save some megabytes of RAM per server).

[Zaish]

We try and use only the S+ version of items (https://ark.wiki.gg/wiki/Mod:Structures_Plus/S%2B_Dedicated_Storage). Also 1 S+ item aggregator for the S+ Vivarium and another to move metal to forges(doesnt seem to work for the S+ Tek Forge) and crops.

I saw in the document that there was a setting to use UseOptimizedDediStorage. I couldnt find out what this did, all the documents it linked to was removed. This setting is still set to false. It sounds like UseOptimizedDediStorage might enable the change you mentioned. I havnt been able to locate any documents that specify what this setting did.

Instead of using a mod to increase the stacking size of items, I bumped up the stack size multiplier built in to 3x. Also added a few stack size rules for prime meat and mutton. I thought we might be storing to much material but we are only using 1700 out of 10000 slots available in our biggest dedicated storage.

Configuration can be found here:
https://pastebin.com/n4j6b7x6

[Scramble]

I mean in S+ there are 2. Exact names:
S+ Dedicated Storage
S+ Tek Storage

The stack change* in Dedicated Storage is done by Wildcard in the ARK game itself and on server - I do not know what impact it had on server-client interface, just staying clear as possible because it is one possible cause of errors.

I do not crash having a few of the S+ Dedicated Storage (I only just got the Engram unlocked so only have a few). Specifically I have 2 nonmod Dedicated Storage Element Shards in both** (crate drops from before engram) and 5 S+ Dedicated Storage with Fiber, Thatch, Wood, Stone, Flint. All 7 near each other, but only touching their own type 2 and 5.

I have not used the Vivarium or Aggregator. The aggregate command in the S+ storage I can use without crashing. But it may also depend on what you pull and what other structures are near. Just like the Vivarium can possible be the issue depending what eggs you get near.

I do not have UseOptimizedDediStorage in GameUserSettings.ini, the Dedi options I do have are:
(grouped)
DedicatedStorageAllowNonStackables=false
DedicatedIntakeMaxRangeInFoundations=50
DediInterfaceDinoBlacklist=
DediInterfaceDinoWhitelist=
DediInterfaceMinUploadInterval=10
(and by itself)
DedicatedStorageSlotCount=10000


But all this text just to say, I can use the S+ Dedicated Storage (Genesis 2, not 1), don't know if the other can be a problem.


* https://survivetheark.com/index.php?/forums/topic/636218-community-crunch-284-dedicated-storage-change-evo-event-and-more/

**The first stopped at 300000 shards once, so I made the second contain Shards too thinking it was full. But now they are 19968 and 313096, I just link Stryder to the one when harvesting space, then occasionally empty some down to the other, not knowing what the max actually is.

[Zaish]

Thanks for the info. I am wondering if that may be what is affecting me. I thought it might have been the S+ propagator, but it doesn't look like anyone used it. I have 72 S+ Dedicated Storage, 1 box per resource. The biggest item is fiber and thatch at 1.5M but when I hover over the boxes, they only show 1700 out of 10000 slots in use.

**Scratch the below. Everything worked fine until a world save occurred. I will try splitting up or dumping some of the resources to see if this changes anything for me.

----Whats odd is everything works as it should when someone is online. As long as my base doesn't enter stasis (my theory) than no one crashes. I am not sure if this affects saving on the back end, meaning if I restarted the server it would load again. For this particular issue, I had to revert back to the last time someone visited the base. Everyone transferred off to do something on the other maps, which meant I used the save done 18 hours ago. A more recent save worked with SS but an older save was needed to get s+ working again.

I read that you can prevent stasis by setting a dino on wonder on the hitching post. I attached someone to see if this helps. I also stopped the S+ Aggregator from collecting more fiber. I mostly use the Vivarium to collect element shards. I have two and they mostly contain clones griffons, managarmr, and recently added gachas. I had a few gachas on the S+ Gavager but I boxed those up to see if they were causing the crash.----

[Zaish]

Scramble,
Your line of thought make sense. I reverted back to an older save and removed all 72 S+ Dedicated Storage boxes to rule it out. I thought things were back in order but a few hours later it crashed. I got a slightly different message this time around that I can research at least.

Do you use the AllowIntegratedSPlusStructures setting?

Fatal error!

VERSION: 358.17
ShooterGameServer.exe!FHangThreadWatcher::HangeDetected() (0x00007ff6b76fa1f0) + 0 bytes [f:\build\lostisland\engine\source\runtime\engine\private\gameengine.cpp:172]
ShooterGameServer.exe!FHangThreadWatcher::Run() (0x00007ff6b76fa143) + 0 bytes [f:\build\lostisland\engine\source\runtime\engine\private\gameengine.cpp:136]
ShooterGameServer.exe!FRunnableThreadWin::Run() (0x00007ff6b7131caf) + 0 bytes [f:\build\lostisland\engine\source\runtime\core\private\windows\windowsrunnablethread.cpp:73]
ShooterGameServer.exe!FRunnableThreadWin::GuardedRun() (0x00007ff6b7131ba8) + 8 bytes [f:\build\lostisland\engine\source\runtime\core\private\windows\windowsrunnablethread.cpp:26]
KERNEL32.DLL!UnknownFunction (0x00007ffc77777344) + 0 bytes [UnknownFile:0]
ntdll.dll!UnknownFunction (0x00007ffc78ae26b1) + 0 bytes [UnknownFile:0]
ntdll.dll!UnknownFunction (0x00007ffc78ae26b1) + 0 bytes [UnknownFile:0]
Last Replicated Actor PlayerPawnTest_Male_C_5
Last Remote Function InitializeState_Multicast

[Zaish]

Following up with this post.

We left Genesis 1 thinking something was corrupted on the map. The issue followed us to the Island where we moved the propagator. I found that the issue could be replicated if I only used the 731604991 - S+ plugin on both The Island and Genesis 1. This may be a bug, or I exceeded an unknown limit on the ini settings. When I setup the cluster a few months ago I changed the default PropagatorFuelInterval=86400 to 172800(3 days) to make it cheaper for everyone to use when we got to a point it was unlocked.

When we first used the Propagator, it showed the correct time in the UI and worked as expected. After some unknown time, it would cause anyone who rendered the base after it was in stasis for a bit to crash the whole server. After removing PropagatorFuelInterval, no more crashes have been occurring. Genesis 1 save that I thought was corrupted worked again.

S+ does not have a discord\discussion board. Hopefully this helps someone in the future. I appreciate everyone help to narrow down the issue.

[Scramble]

Originally posted by Zaish:

I read that you can prevent stasis by setting a dino on wonder on the hitching post.

Do you use the AllowIntegratedSPlusStructures setting?

I recently done some tests with the Hitchpost. It is only the leashed dino that is kept out of stasis, the entire base still enters stasis. If you use a Maewing to feed kid dinos, each kid dino needs leashed, leashing the Maewing is not enough. I am guessing the Maewing as trough will work without leash, but I did not test that.


S+ does have discussions, Steam Website ASE section: (line immediately below "ARK: Survival Evolved") Workshop -> (line below the icon picture of girl with Assault Rifle and Rocket Launcher on a Raptor) -> Discussions


I am sorry I missed the question, I see you found the culprit, but for anyone messing around with this in future - here is my setting.

AllowIntegratedSPlusStructures=True

Same for all servers I checked. As I remember it, it is a must for something I use and has always been on for any server I ran with S+ on it.


Good style posting your findings.

[Zaish]

Originally posted by Scramble:

Originally posted by Zaish:

I read that you can prevent stasis by setting a dino on wonder on the hitching post.

Do you use the AllowIntegratedSPlusStructures setting?

I recently done some tests with the Hitchpost. It is only the leashed dino that is kept out of stasis, the entire base still enters stasis. If you use a Maewing to feed kid dinos, each kid dino needs leashed, leashing the Maewing is not enough. I am guessing the Maewing as trough will work without leash, but I did not test that.


S+ does have discussions, Steam Website ASE section: (line immediately below "ARK: Survival Evolved") Workshop -> (line below the icon picture of girl with Assault Rifle and Rocket Launcher on a Raptor) -> Discussions


I am sorry I missed the question, I see you found the culprit, but for anyone messing around with this in future - here is my setting.

AllowIntegratedSPlusStructures=True

Same for all servers I checked. As I remember it, it is a must for something I use and has always been on for any server I ran with S+ on it.


Good style posting your findings.

Are you referring to this? https://steamcommunity.com/workshop/discussions/?appid=346110. Thanks for pointing this out, I should have created this discussion under the mod sub. Some of the mods have their own dedicated discussion\discord section under their workshop. S+ used to have one but it looks like it was locked at some point in the past. At least I cant find it and Google searches lead me to a locked discussion board.

[Scramble]

Originally posted by Zaish:

Are you referring to this? https://steamcommunity.com/workshop/discussions/?appid=346110. Thanks for pointing this out, I should have created this discussion under the mod sub.

Yes, apologies. Not a dedicated S+ area, but it is semi active for ASE mods.

Not trying to say you posted in the wrong section. It wasn't really narrowed down all that much when you wrote the first post.