Offloading AI Processing - Increasing server performance for Arma III and the coming issues

[Mirudes]

(From zorrobyte?)
I have copied and pasted this from "Google Drive - Google Docs" because this is well worth a read.

Offloading AI Processing[docs.google.com]
Increasing server performance for Arma III and the coming issues


What is the Arma server?

An Arma server is a program that handles packets between connected clients while also being a client within itself.
Each client could also be considered a server with the exception that they do not directly send packets to other clients. They send packets to the server to be rebroadcast to other clients.
For example, if I have 3 guys in my squad my CPU is handling the simulation of these characters (the CPU load). My client then sends the movements, actions, etc to server which is then rebroadcast to every connected client. I tell my guy to move 100m, my CPU does the work and other players see my guy move via packets sent over the network.


Arma server broken down

The Arma server has a lot of work to do on it’s own. It has to run anti-cheat, broadcast what players are connected to the multiplayer server list, handle VoIP and chat, send and receive packets to and from every connected client and much more.
The Arma server also runs a client to simulate the worldspace (map) I move a meter, the server’s copy of the world moves me a meter then it sends that I moved a meter to all connected clients making me move a meter on everyone else’s screen. This is visually observable when a connected player has a bad connection and you observe the player and the player’s AI or vehicle he’s driving/piloting skipping around across your screen.


The anatomy of a mission

In most missions you would place a unit or groups of units on the map and give them waypoints using the Arma editor.
You set up triggers, markers and other essential items to take action during the mission.
Some may choose to add in scripts to their missions such as revive or ammo box refiller scripts. Some may add many scripts or use scripts entirely to place units, triggers, markers, etc.
There is no standard method of creating an enjoyable mission as there are many ways of accomplishing the outcomes either via scripts or editor


AI: The downfall of server performance

In exception to 120+ player PvP missions in which netcode (the methods of relaying packets between clients) may present such a load as to “lag” the server; most scripts, triggers, markers and other mission elements do not significantly lag the server, unless of course they are badly designed or have bugs.
AI or Artificial Intelligence poses the most load to the server as the name would suggest. All AI that is within the mission are simulated within the Arma server (of course differs with special circumstances, HC etc; continue reading)
With hundreds of AI the server begins to “lag” up to the point the server becomes so “laggy” it’s unplayable.


What is “lag” in Arma?

Much like when playing games on your computer, the server also has FPS or frames per second.
FPS in the server simulation is not how fast the graphics render as it has none, it’s how many times per second the simulation is updated (basically, I know there is an inverse relationship etc, out of scope of this doc)
“lag” as in the simulation is most generally when the server FPS drops below 15 FPS although some may tolerate the server until it drops to 5 FPS.
AI count (how many) is the quickest and sure way to impact server performance as it inherently presents the most load.


Why not multithreading? It’s easy right?

Well, no. Within the realm of the Arma server you simply cannot have out of order processing within the same worldspace. If you simply split AI into it’s own CPU core it will accomplish nothing and have weird results. Take a math problem, if you have a logical order of problems and attempt to jump ahead and solve the second part before the first, it'd be impossible, at least if trying to do it all at once or within the same context.


So what can you do?

Well, lots of things; some of which are simple, others hard. Some rely on BIS (Arma’s Developers) to implement or maintain.
One is caching in which the AI are deleted and respawned when the players get close enough or their processing is paused until yet again, the player gets close enough; With this method you can also hide the actual model of the unit to reduce graphical load.
Another is using scripts to not spawn all the AI in at once so only a limited number are active during any given time.
Another is offloading the AI onto another process entirely so the server can simply focus on relaying packets…


Wait, multithreading? You said it was impossible!

In this document I shall refer to the HeadlessClient as (officially named) “Dedicated Client”
The Dedicated Client is a client in which does not render graphics and sole role is to handle AI load for the server. This way the server can focus on handling anti-cheat, packets, VoIP, etc and not be slowed by AI.
Even if there are hundreds of AI on the Dedicated Client and it can only simulate 5FPS, the server itself will not be effected hardly as much as it has a much reduced workload (with certain exceptions such as weird netcode behavior, etc in extreme cases).
In cases the DC is getting 5FPS you may see the AI on the DC skip around but not other players or player’s AI, etc.


Dedicated Client

More specifically the Dedicated Client is much in appearance as to the dedicated server as it’s a window with text (a console window).
Although it may appear to be more “server like” than a normal client, the Dedicated Client is nothing more than a regular game client without a graphics renderer. It still sends packets to the server to be rebroadcast but instead of the Dedicated Client only processing AI in it’s squad as with a normal client; scripts are used to spawn all AI within the Dedicated Client’s worldspace.
The dedicated client has a name and even uses a playerslot in the server just like any other client.


How do I use DC?

Simply put a script that spawns AI can made to spawn on the HC by adding if !(player == DCSlot) exitwith {}; to the start of the script and placing a playable unit called “DCSlot”; then connecting and moving the DC into the slot when starting the mission.
An added bonus is not only can you have more AI, few think about actually needing less AI as the AI can simulate faster making them deadlier further helping performance
I could write a whole document on the technicalities of using the DC but will leave this up to research of the reader. It’s not terribly difficult once you figure it out for the first time.


The issues surrounding DC

It isn't rocket science but if you are used to only using the editor you may become frustrated trying to use DC as you must use scripts to spawn AI
You have to buy a second copy of the game at minimum or many if you wish to run more than one DC per server.
In Arma 3 there is issues around Steam and CdKeyChecking. There are solutions such as SteamCMD and buying yet more copies of the game which further push DC into obscurity via complexity. DC isn’t even officially documented so further obfuscation may push DC into oblivion and eventually be dropped all together.


The debate

Currently there is a debate over the whole situation of having to buy another copy of Arma. Some regard the issue as no big deal while others think DC should be part of the server or at least included in their purchase of Arma.
Some may claim that releasing a DC that allows a customer to run a DC without having to buy another copy poses a security risk due to numerous reasons; some disagree but who knows truly other than BIS as we don't have source code although our logic is simple…


How you can help

We have a voice via the Arma 3 feedback tracker, vote on this issue if you understand the topic http://feedback.arma3.com/view.php?id=7082 (*)
If you have development knowledge then please post your feedback and proposed solutions within the ticket
If you are an aspiring mission developer that’s looking for more performance then look into how DC works and how to use it yourself (search for Arma Headless Client as the community refers to DC as HC)
The possibilities are limitless. A framework could automatically assign AI to the DC if server FPS drops below 15, etc etc; All within scripting and could be community made.

(*) 0007082: Newest CD key checking breaks HC

[Pèpè Silvia]

very informative, thank you! It's just a shame that 99% of the people in these forums probably won't even read/understand this :(

[grent]

Seems like that would solve a few problems, looking into that option today. Thanks pal.

[Dwarden]

well i would suggest you try use 1.14hotfix PERF1 server experimental binary
http://forums.bistudio.com/showthread.php?169944-Arma-3-STABLE-server-1-14-quot-performance-binary-quot-feedback

it has AI specific performance tweaks

note: the OP and the document linked has quite some misconceptions and misunderstanding originating from the 'guess' approach of authors

[Zorrobyte]

Thanks for the repost to steam of my document. Sure, there may be some misconceptions but only so much is possible without engine source code and documentation as to internal AI processing. I'd be more than happy to revise my document for accuracy with your help Dwarden <3

Love and kisses,
Zorrobyte

[NZPorterDJ]

Isn't the server executable able to run a dedicated client without the need to buy another copy of ArmA 3? Or is that dev branch only?

I could be miss-remembering, though.

[Dwarden]

zorro you realize you necroed year old obsolete post ?