seg-fault on linux-server

[CryptEarth]

server-version: idk - but its compatable with current 5.01 client
thats what "status" gives: "version : 48/5.0.0.0 7036 secure (225840)"
in server-browser it shows "5.01"

assert_20160201232232_1.dmp[11789]: Uploading dump (out-of-process)
/tmp/dumps/assert_20160201232232_1.dmp
assert_20160201232232_1.dmp[11789]: Finished uploading minidump (out-of-process): success = no
assert_20160201232232_1.dmp[11789]: error: libcurl.so: cannot open shared object file: No such file or directory
assert_20160201232232_1.dmp[11789]: file ''/tmp/dumps/assert_20160201232232_1.dmp'', upload no: ''libcurl.so: cannot open shared object file: No such file or directory''
./svends_run: line 255:  3965 Segmentation fault      (core dumped) $GameBin_CMD

first: this is a linux64 - but I didn't had installed libcurl-32 yet - that's the cause of minidumb-fail
side-note: you should add to your server-guide that admins runing linux-64 have to install libopenssl-32 and libcurl-32 ~ maybe some other libs that will show up in the future
second: yes - this server is running metamod-p (sc5 version from sc-forum) with amxx 1.8.2 + scxpm - I'll now try to run it w/o metamod enabled cause I think thats the main-cause
currently my server is running open for the community - and some players got already lv 300-400 (maybe some xp-exploits as getting so high would take weeks or even months ... but not in 48 hours) - this will be a surprise if scxpm is disabled ... I have to add a note

if anyone wants to play on my server - look out for a server named "[DE/FR] cryptearth.de SC5 community server" - or join by domain : cryptearth.de (pls don't use IP as it can change over time)

[CryptEarth]

update
ok - so far the server still runs stable scince I started this topic
I will provide more information if the server is still up after a full 24h
if so at least metamod or something further down the line might had caused the seg-fault
as libcurl-32 wasn't installed as the seg-fault occured I don't know what would had happened after it - maybe I will force it to the next seg-fault
maybe some needed info: after the server crashed in the seg-fault htop showed me it hung up on a call of "cat" - so it tried to log something but can't cause of missing libcurl-32
if the server makes the full 24h I will let it run for another 24h so it passes full 48h - if it's stable I will add metamod and try another full 48h run
if it keeps stable it goes further down to amxx w/o scxpm - and if this is stable I try to figure out what will happen if scxpm crashes again

[CryptEarth]

well ... while checking server status I noticed it just froze and hung up with 100% cpu load
so this seems to be a problem by the current vanilla linux server itself

[CryptEarth]

well - really strange
sometimes it just runs for a few hours - like last time - and then hangs up if full cpu load (i guess some sort of deadlock and infinite loop (pls add some debug-option and log-outputs so it's visible where it gets stuck)) - and there some other times it runs just fine for over 24h continously
so as there was no seg-fault when running vanilla - i step up and add metamod (w/o amxx for now) and see if this results in a seg-fault
really want to track this down to the cause

[Tunac]

I'm currently investigating a segfault on 155.94.222.5:27020 with a debugger running. Hopefully we can find out the cause soon.

[Elij4h]

I have similar troubles, getting a segfault when connecting to the server.
The server segfaults directly when connecting.

The server is a ubuntu 14.04 LTS 32bit machine (KVM), DNAT.
gdb output of the segenv:

ortwin@ubuntu32:~/steam/steamapps/common/Sven Co-op Dedicated Server$ ./svends_run 
Auto-restarting the server on crash
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./svends_i686...done.
(gdb) start
Temporary breakpoint 1 at 0x8049100: file ../dedicated/sys_ded.cpp, line 966.
Starting program: /home/ortwin/steam/steamapps/common/Sven Co-op Dedicated Server/svends_i686 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

Temporary breakpoint 1, main (argc=1, argv=0xbffff6d4) at ../dedicated/sys_ded.cpp:966
966	../dedicated/sys_ded.cpp: No such file or directory.
(gdb) continue 
Continuing.

Console initialized.
Using breakpad crash handler
Setting breakpad minidump AppID = 225840
Forcing breakpad minidump interfaces to load
Looking up breakpad interfaces from steamclient
Calling BreakpadMiniDumpSystemInit
Protocol version 48
Exe version 5.0.0.0 (svencoop)
Exe build: 17:35:04 Jan 29 2016 (7036)
STEAM Auth Server
couldn't exec language.cfg
Server IP address 127.0.1.1:27015
Can't register variable servercfgfile, already defined
Can't register variable lservercfgfile, already defined
Can't register variable mapcyclefile, already defined
Can't register variable motdfile, already defined
Server logging data to file logs/2016-02-06.log
L 06/02/2016 - 22:48:24: Log file started (file "logs/2016-02-06.log") (game "svencoop") (version "48/5.0.0.0/7036")
L 06/02/2016 - 22:48:24: Server cvar "sv_password" = "***PROTECTED***"
L 06/02/2016 - 22:48:24: Server cvar "sv_aim" = "0"
couldn't exec banned.cfg
couldn't exec listip.cfg

Program received signal SIGSEGV, Segmentation fault.
0xb6bb8568 in BSecure (this=<optimized out>) at ../engine/sv_steam3.h:72
72	../engine/sv_steam3.h: No such file or directory.
(gdb) bt
#0  0xb6bb8568 in BSecure (this=<optimized out>) at ../engine/sv_steam3.h:72
#1  Steam_GSBSecure () at ../engine/sv_steam3.cpp:1033
#2  0xb6b87325 in SVC_GetChallenge () at ../engine/sv_main.c:3272
#3  0xb6b90a96 in SV_HandleRconPacket () at ../engine/sv_main.c:9114
#4  SV_CheckForRcon () at ../engine/sv_main.c:9205
#5  0xb6b5807b in _Host_Frame (time=time@entry=0.00117714598) at ../engine/host.c:1412
#6  0xb6b58298 in Host_Frame (time=0.00117714598, iState=1, stateInfo=0xbffff49c) at ../engine/host.c:1526
#7  0xb6b80304 in CEngine::Frame (this=0xb6c4c420 <g_Engine>) at ../engine/sys_engine.cpp:245
#8  0xb6b7ca45 in CDedicatedServerAPI::RunFrame (this=0xb6c47a60 <__g_CDedicatedServerAPI_singleton>) at ../engine/sys_dll2.cpp:1315
#9  0x08049b92 in RunServer () at ../dedicated/sys_ded.cpp:772
#10 0x08049372 in main (argc=1, argv=0xbffff6d4) at ../dedicated/sys_ded.cpp:1152
(gdb) info sharedlibrary
From        To          Syms Read   Shared Object Library
0xb7fde860  0xb7ff67ac  Yes         /lib/ld-linux.so.2
0xb7f2c9f0  0xb7f9acdd  Yes (*)     ./libstdc++.so.6
0xb7edcad0  0xb7edd96c  Yes         /lib/i386-linux-gnu/libdl.so.2
0xb7ec3840  0xb7ed04a7  Yes         /lib/i386-linux-gnu/libpthread.so.0
0xb7d28420  0xb7e59b6e  Yes         /lib/i386-linux-gnu/libc.so.6
0xb7ccf600  0xb7cfd70b  Yes         /lib/i386-linux-gnu/libm.so.6
0xb7cb7e04  0xb7cc7490  Yes         ./libgcc_s.so.1
0xb6b3b130  0xb6bc97f1  Yes         /home/ortwin/steam/steamapps/common/Sven Co-op Dedicated Server/engine_i686.so
0xb6ade5c0  0xb6ae8d74  Yes (*)     ./libsteam_api.so
0xb6ad49b0  0xb6ad80ca  Yes         /lib/i386-linux-gnu/librt.so.1
0xb6abbe20  0xb6ac8698  Yes         /home/ortwin/steam/steamapps/common/Sven Co-op Dedicated Server/filesystem_stdio.so
0xb595fd00  0xb65de780  Yes (*)     /home/ortwin/steam/linux32/steamclient.so
0xb586f130  0xb5897064  Yes (*)     /home/ortwin/steam/linux32/crashhandler.so
0xa5859ad0  0xa58606bb  Yes         /lib/i386-linux-gnu/libnss_files.so.2
0xa51859c0  0xa55bc325  Yes (*)     /home/ortwin/steam/steamapps/common/Sven Co-op Dedicated Server/./svencoop/dlls/server.so
0xa4f4b1a0  0xa4fc8d70  Yes (*)     ./svencoop/dlls/libsqlite3.so
0xa4eff5a0  0xa4f2ad78  Yes         ./svencoop/dlls/libmariadb.so.2
0xa4ee2750  0xa4eef1d4  Yes         /lib/i386-linux-gnu/libresolv.so.2
0xa4df63a0  0xa4e11684  Yes (*)     ./libiconv.so.2
0xa4da8160  0xa4de01b4  Yes (*)     /lib/i386-linux-gnu/libssl.so.1.0.0
0xa4c27b40  0xa4d20764  Yes (*)     /lib/i386-linux-gnu/libcrypto.so.1.0.0
(*): Shared library is missing debugging information.
(gdb) 

[CryptEarth]

it's really wired - now it ran more than 48h in one go (forgot to monitor sometimes - but set -norestart flag) without any issues
so now i enabled metamod (with empty plugin list so amxx will not be loaded) and see if its keep stable

also another problem - my server has to update - but as some package-dependencies broke libcurl can't be longer provided - i have to figure a solution

[CryptEarth]

and there we go again - another seg-fault - just after adding metamod and playin about half-way through hl-sp (right before the mined building)
so i guess metamod could be a possible reason on my system - but it only said three

crash
wrote minidump
upload

and the upload line said some like "discarded=1" - so i don't know if this what curl is used for or if you even received the minidump (can make it accessible if needed)
but i didn't got such a nice listing as Eli got
so i will remove metamod and see how it performs again

[Jazz]

I'm having problem as Elij4h, with same backtrace

Server is under Ubuntu 15.10, 64-bit

coredump: https://my.mixtape.moe/enyfkw.tar.gz

[CryptEarth]

well - maybe a helpfull information : my server is running opensuse tumbleweed / based on 13.2
as long time passed since - first time i tried to setup a sc server (i think it was back in the 4.0/4.5 days) i also had heavy issues with the opensuse os
may it's just one of those distros not good for steam server hosting

[Jazz]

Ok so I've read the guide here paying extra attention: http://www.svencoop.com/manual/server-basic.html
I didn't set +map parameter so that's why the server crashed on start. I know it's written in the guide but it would be nicer to get an error message instead of a cryptic segfault.

Sen, if you want to get a better error message, try running svends_run with -debug
Another option is to set

ulimit -c unlimited

before running the server, run it with -norestart, make it crash, and it will generate a core file. You can send the core file to devs or post the backtrace here. Maybe it's the same problem they're currently trying to fix:

Originally posted by Dynamite:

I'm currently investigating a segfault on 155.94.222.5:27020 with a debugger running. Hopefully we can find out the cause soon.

To get the backtrace from core file:

$ gdb ./svends_i686 ./core
(gdb) bt
< backtrace should be printed here >
(gdb) quit

[CryptEarth]

@jazz
Ah, thx for the info. Didn't know about the debug option. Also I will try to lookup the backtrace.

[Elij4h]

Thanks for the tip with the +map parameter!
Server is running now, will report back if issues arrive at a later time!

[CryptEarth]

hmm - currently with metamod enabled - it runs pretty stable
maybe I'll try to add amxx this weekend

[CryptEarth]

ok - here is my progress so far

vanilla - run pretty stable - here and there some "hung up" - maybe due to bad code like race conditions or endless loops (like buffer overflows and such) - but at stress-test (full 12-16 player over 48h+) no crash - STABLE
metamod - the same - no noticeable difference - and after fixing libcurl-32bit (just took my distro a few days to catch up) now prepared for crashing again - STABLE
amxmodx - currently testing over the weekend and next week - currently in mysql-mode for admin-access - later for sharing XP for scxpm over multi-server (want to set up 4 servers - vanilla (w/o hl-sp), HL only, OpFor only, BS only) and access on website (like mpuksc)

still some major and some minor configs to be made - i would also wish mpuksc would release thier fixed version of scxpm - maybe try to bother those guys gently again

if amxx itself makes issues - maybe have to look up for some newer versions (as 1.8.2 is as old as HL itself - well ... almost)
if it runs fine so far i'll try add scxpm again

also i'll try to add EntMod - just cause it's a powerfull admin-tool

[CryptEarth]

and another seg-fault about 30min ago
even i have -debug option enabled and gdb installed i don't got a core-file to get a backtrace with gdb
only i got is a dmp file in /tmp/dumps/ wich is kinda binary