[Rust][Dedicated] 10-30 Minute Freeze-ups

[Herb]

I am running my Rust server on a fresh Debian 8.5 install, on a dedicated box that runs literally nothing but one Rust server via LGSM, along with New Relic monitoring.

Everything runs smoothly for 8+ hours, then the server lags out and becomes unreachable for anywhere from 5-30 minutes. After the spike, everything bounces back to normal.

I have nearly ruled out network issues being the culprit, because I have run MTR reports during these times of lag and everything looks perfect. I am able to SSH to my server, and if I kick off a reboot everything comes back just fine (besides crafting queues).

I have been struggling with this issue for some time and would love some advice.

The lag:
Everything will be going fine, then actions will stop registering. You'll stop getting wood for hitting trees, you cannot open doors, etc. The server doesn't go down, the process never crashes, but nothing registers. It just hangs. Then in 5-30 minutes the server will bounce back and everything will go back to how it was when the lag hit. Rarely, the server will enter repeated loops of these spikes for an hour or two, which really hurts.

Server specs:
-Intel E3 1240 V3
-Samsung EVO-850 256gb SSD
-2x 8gb DDR3 1333 ECC

Software:
-Oxide w/ XpBooster plugin (plugin is very simple, written by creator of Oxide (Wulf)
-This issue has been occurring since before we installed Oxide

Ulimit:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128216
max locked memory (kbytes, -l) 128
max memory size (kbytes, -m) unlimited
open files (-n) 256512
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 16386
cpu time (seconds, -t) unlimited
max user processes (-u) 128216
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[Mazer]

If it's repeatable, fire off vmstat in another console or adjacent tmux session to see what resources are thrashing when this happens.

[Herb]

Originally posted by CedarLUG:

If it's repeatable, fire off vmstat in another console or adjacent tmux session to see what resources are thrashing when this happens.

So to confirm, during a lag spike, you'd like me to SSH in as the user that controls RustDedicated (the one I installed LGSM with) and simply run "vmstat"?

Just want to make sure I have the process right, because the issue only happens once a day or so, so if I don't nail it the first time I won't have another chance for a while.

edit: running vmstat during a time of normal operation (right now, no lag spike) returns this:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 12983608 6492 180800 0 0 33 10 987 734 8 1 92 0 0

[Mazer]

Originally posted by Herb:

So to confirm, during a lag spike, you'd like me to SSH in as the user that controls RustDedicated (the one I installed LGSM with) and simply run "vmstat"?

No. What I'm suggesting is to run it for hours. Start it under a separate tmux session, or inside of Rust's tmux terminal ("Ctrl-b c" creates a new console inside of tmux. "Ctrl-b n" takes you to the 'n'ext existing console, "Ctrl-b p" takes you to 'p'revious console within the same tmux session).

Start up `vmstat -n 1 | tee /tmp/vmstat.out` when you start Rust. Then when the system freezes up, you can correlate the system resources with the vmstat output at the same time.

If you want more detailed time integrated into the log, then try something like:

for ((;;)) ; do vmstat  2>&1 >> /tmp/system-status.out; date >> /tmp/system-status.out; sleep 1; done

Then when your system freezes, note the time and correlate it with the vmstat output in /tmp/system-status.out.

[DrMohn]

Having the same probleme here... Random Freeze-ups.

The ressource manager shows that rust stops using any ressources at all.

Any help?

We are using an Ubuntu v-server

[Mazer]

"The resource manager ...". <- Please elaborate.