Changes to server browser packets in latest Steam client

[FletcherDunnValve]

Hello!

The Steam client released today includes changes to the client intended to address a gameserver reflection attack vulnerability in the server browser protocol. It also includes several experimental features that can be enabled on gameservers to further harden the server against attack.

When a client queries a server, it will begin by sending an A2S_INFO packet, formatted exactly as before. If the server desires (see below for how to opt in), the server may reply with a S2C_CHALLENGE challenge request, instead of replying with the S2A_INFO_SRC data packet. The new client understands this reply, and can resubmit the A2S_INFO request, authenticated with the server's challenge, thus proving that it is not spoofing the source IP address.

The binaries released with the client today also include changes relevant to how the gameserver handles these packets. First, challenge generation was hardened against attacks designed to fill the challenge table and crowd out legitimate challenge requests.
Additional experimental improvements are available by setting environment variables, but are not enabled by default:

• STEAM_GAMESERVER_A2S_INFO_REQUIRE_CHALLENGE=1. This will opt into the new handshake for A2S_INFO packets to address the reflection attack. At this time, it is primarily intended for testing 3rd party query clients. Since not all players may have an updated Steam client and understand this handshake, it is not recommend to enable this at this time, except for testing. We will post again when the vast majority of users are running clients that understand the new protocol, and enabling the new protocol is safe.
  
  
• STEAM_GAMESERVER_RATE_LIMIT_200MS=N. This will drop any connectionless packets (A2S_INFO, A2S_RULES, A2S_PLAYERS) from a given IP after more than N are received in a 200ms window. By default, rate limiting is disabled, but a reasonable value might be somewhere around 25-75.
  
  
• STEAM_GAMESERVER_PACKET_HANDLER_NO_IPC. If this variable is set, then the Steamworks packet handling calls will use a “fast path”. In the Steamworks SDK, most API calls are serialized over an IPC, and in fact execute in the steam client process. A dedicated server does not communicate with a steam client (there usually isn’t one running), but the same basic design is used – there are two threads, and all API calls are serialized and executed in the Steam thread. Turning on this environment variable will bypass this serialization and thread context switch, which makes these calls, much, much faster. Note that this only affects a server in “shared socket mode” – meaning the game port and the query port are the same.

Gameserver operators can obtain binaries for all platforms, including older linux distributions, here:
http://media.steampowered.com/apps/steamworks/steam_bins_6246211.zip

These changes were previously discussed in this post: https://steamcommunity.com/discussions/forum/14/2989789048633291344/.

[76561199083226126]

bloody epic!

[Jessie]

Originally posted by FletcherDunnValve:

• STEAM_GAMESERVER_RATE_LIMIT_200MS=N. This will drop any connectionless packets (A2S_INFO, A2S_RULES, A2S_PLAYERS) from a given IP after more than N are received in a 200ms window. By default, rate limiting is disabled, but a reasonable value might be somewhere around 25-75.

How long does this last ? or is it just the next 'x' packets are dropped after 'N' within 200ms ?

eg: N=25
after 25 packets are received within 200ms, any more packets are dropped until the 200ms window is over ?

or

after 25 packets, all future packets are dropped for ?? seconds ?

[BONGOMBAWAGAMBA]

what does it all mean im confused!

[PredatH0r]

I'm the author of https://github.com/PredatH0r/SteamServerBrowser and I just heard about this thread here. I'm also a modder and game server owner for TOXIKK, a UDK/UE3 based game that uses UE3's wrapper around the Steamworks API for server queries.

Can you please clarify what games will be affected and how changes are going to be pushed to the servers?
Is this for Source engine only (as this thread is in the Source Dedicated Server forum) or does it affect all game servers that use the Steamworks API dlls (i.e. Quake Live, UT3, TOXIKK, ...)?
Will game developers have to release updates for the game?
Will server owners, when they run updates for their games, receive the modified Steamworks API dlls without the game developers releasing changes for the game itself?

Are there any example game servers (IPv4:port) that already use this challenge protocol so I can test my SteamServerBrowser client against it?

[PredatH0r]

You mention an S2C_CHALLENGE and S2A_INFO_SRC packet in your initial post here. Is this documented anywhere?
I can't find anything about it on https://developer.valvesoftware.com/wiki/Server_queries

[PredatH0r]

@playboy31.fr:
Where did you get the information about an 8 byte challenge?
For the A2S_RULES and A2S_PLAYERS the server sends a response with a 4 byte challenge:
FF FF FF FF 41 xx xx xx xx
(as documented in the link above in my previous post)

I assume that for the new A2S_INFO the same packet format is used for the challenge.

[Robin]

As of the client update this morning all our LAN-servers have turned invisible and aren't returned in searches. Is this a known side effect? Is there some change we need to make on our side that is required by this update?

[FletcherDunnValve]

Originally posted by Jessie:

How long does this last ? or is it just the next 'x' packets are dropped after 'N' within 200ms ?

eg: N=25
after 25 packets are received within 200ms, any more packets are dropped until the 200ms window is over ?

Any packets recieved beyond N, within the 200ms window, are dropped. When the 200ms window ends, packets are received again. The design goals of this system are: 1.) Limit the ovberall rate. 2.) Allow for some "bursty" behaviour. 3.) Make sure implementation is not vulnerable to attack where very large number of packets from many different (spoofed) source addresses are sent.

[FletcherDunnValve]

Originally posted by PredatH0r:

You mention an S2C_CHALLENGE and S2A_INFO_SRC packet in your initial post here. Is this documented anywhere?
I can't find anything about it on https://developer.valvesoftware.com/wiki/Server_queries

I've updated the wiki.

[FletcherDunnValve]

Originally posted by PredatH0r:

Can you please clarify what games will be affected and how changes are going to be pushed to the servers? Is this for Source engine only (as this thread is in the Source Dedicated Server forum) or does it affect all game servers that use the Steamworks API dlls (i.e. Quake Live, UT3, TOXIKK, ...)?

To be clear, the change today only makes the client *capable* of speaking the new protocol. If the server is not updated, the packets sent and received will be exactly the same as before! Testing revealed that this was very important, as there are some extremely aggressive filters in use in the wild, and any change to the protocol that is not initiated by the server is likely to trigger those filters.

By "client", I mean all games that use the Steamworks ISteamMatchmakingServers interface. This is implemented in steamclient.dll, which is part of the Steam client, and is NOT shipped with games. So, when users update their steam client, they will be updated to speak the new protocol. Within about 2 weeks, you should expect the vast majority of users (>95%) will have the new client.

Any clients that manually craft UDP packets will need to update their code to be able to respond if a server responds to A2S_INFO with a challenge, and repeat their A2S_INFO request with the challenge appended. (NOTE: The ping should be calculated carefully in this case. It must be calculated based on the RTT of A2S_INFO and the S2C_CHALLENGE *only*. Do not recalculate it based on the second request, when the S2A_INFO_SRC arrives, or else the server can do bad things to fake a better ping than they really have!) I took a look briefly at your githjb project, but I couldn't tell if you are crafting UDP packets manually or you are using steamclient.dll to load the packets. If the former, then you probably need to write some new code. If the latter, then you are basically an ordinary "game client" for purposes of this discussion, and it should just work. The only thing you might need to do is update steamclient.dll, etc if you ship those with your app.

At this time, no gameserver changes are being forced! The steam gameserver binaries in the update today and available at the link above (they are the same binaries as the client, but the zip contains also binaries for older linux distros) understands the environment variables detailed above, and you can force the gameserver to opt into the new A2S_INFO behaviour for testing. Because not all users have a client that understands this new behaviour, it's not recommended to turn it on for real servers yet.

Originally posted by PredatH0r:

Will game developers have to release updates for the game?

Not unless they are specifically crafting UDP packets, and wish to remain compatible with servers that choose to opt into the new protocol. The user updating their steam client will automatically upgrade almost all games to understands the new format.

If they are sending these packets through the steamclient.dll that is part of the steam client, *or* they don't care about compatibility with servers that wish to upgrade to the new protocol, they are fine.

Originally posted by PredatH0r:

Will server owners, when they run updates for their games, receive the modified Steamworks API dlls without the game developers releasing changes for the game itself?

The way that dedicated servers get updated binaries depends on the game, because they run in an environment where there is no steam client being constantly updated by the user. Some games (like Valve games) ship a particular version of steamclient.dll with the dedicated server, and so these games will only be updated if the game dev specifically updates the server. We suspect that there will be many dedicated servers who go a long time (perhaps indefinitely) not upgrading to the new protocol. That's OK, although they will remain vulnerable to the reflection attack. Other games add the Steamworks SDK redist (app 1007) into their app. (This is the recommended method -- if you are a dev, please do this if you can. We don't do it ourselves because we often use our own games to test features before releasing them to a wider audience.) So when we update this depot, many games will receive an update at once.

Also -- note that in general, you can almost always just drop in a new steamclient.dll. You do NOT need to upgrade to a whole new version of the SDK or recompile any code! This is what the client does, after all, and the same basic things works for servers, too.

The new server binaries have environment variables that can be used to opt into new behaviour, but if no environment variables are set, no behaviour will change. At some point in the future, we might change the default behaviour. But we will retain a method (e.g. setting the environment variable STEAM_GAMESERVER_A2S_INFO_REQUIRE_CHALLENGE=0) for server operators to opt out, if compatibility with old clients who cannot be updated is important. I'll post again when we change the default behaviour in a way that might introduce the possibility for breaking compatibility with existing clients. We will definitely never push out a server update that we think introduces the possibility of breaking compatibility with a significant percentage of "ordinary" clients.

Originally posted by PredatH0r:

Are there any example game servers (IPv4:port) that already use this challenge protocol so I can test my SteamServerBrowser client against it?

[/quote]

Not yet, because there are still too many clients who don't speak the new protocol. To test your client against a server using the new protocol, you'll need to download a server, drop in the new binaries from the .zip above, and set STEAM_GAMESERVER_A2S_INFO_REQUIRE_CHALLENGE=1.

This might change in a few weeks.

[PredatH0r]

Thanks for the information.

I successfully tested the new protocol with this configuration:
- TOXIKK game server (app id 324810)
- dropped the dlls from the zip you linked above into C:\steamcmd\steamapps\common\TOXIKK\Binaries\Win32, replacing the older DLLs that are shipped with the game
- setting STEAM_GAMESERVER_A2S_INFO_REQUIRE_CHALLENGE=1
- started the server

The Steam Client's built-in browser (View / Servers) and also the game's internal server browser did not show my server before I opted-in to the steam client beta. After beta opt-in the server was listed again in both.

My SteamServerBrowser app constructs UDP query packets directly without steamclient.dll and works now too after a bug fix. I can confirm that the server is sending a challenge as expected.

If anyone needs a server for testing, you can use mine: 138.201.84.93:27113

[Robin]

Originally posted by FletcherDunnValve:

Originally posted by PredatH0r:

Will game developers have to release updates for the game?

Not unless they are specifically crafting UDP packets, and wish to remain compatible with servers that choose to opt into the new protocol. The user updating their steam client will automatically upgrade almost all games to understands the new format.

If they are sending these packets through the steamclient.dll that is part of the steam client, *or* they don't care about compatibility with servers that wish to upgrade to the new protocol, they are fine.

This does not seem to be entirely true though. As the client update rolled out to our computers at our office our developers reported no longer being able to find any LAN servers despite not having opted into this on the server side. Just to be absolutely sure we rolled back all our content and binaries over a week and still saw the same behavior, behavior that right up until the client update worked fine. This might be an unintentional change, but it's really detrimental to our development as we use LAN-servers to test our game server code when developing. Our internet servers however are still reachable and work, it's just that not all devs have access to be able to set up port forwarding for their own internet server. Even when the dedi was on the same machine, LAN searching would consistently not return any servers.

Are you aware of this change? is it being investigated? Is there any way we can roll back this locally in an easy way awaiting a fix?

[PlayBoy31.fr]

@FletcherDunnValve

I have implemented the new query protocol on my side and its ok.

Can you make some adjustement to the query please?

Can you add to the default (first) query \xFF\xFF\xFF\xFF at the end ?

So the first query will be:

\xFF\xFF\xFF\xFF\x54Source Engine Query\xFF\xFF\xFF\xFF")

Then the next query will be for example with challenge DEADBEEF received:

\xFF\xFF\xFF\xFF\x54Source Engine Query\xDE\xAD\xBE\xEF")

Are you ok with this?

[PlayBoy31.fr]

@FletcherDunnValve

The latency displayed in the server window when we do show infos is erroneous, it displayed the latency from first query + latency of the second query with the challenge

So the latency is now like the double (tested on server of me) ...

See the picture: https://pasteboard.co/JE3y5ML.png

tested with the same server (of me) without the patch, latency is normal

[PredatH0r]

IMO the initial request should be left unchanged - as currently planned.
Otherwise there could be incompatibilities with unpatched servers that do not expect any additional data after "Source Engine Query\0".