In-depth game traffic analysis for the devs

[sny]

Hey SNK & other testers!

I work as an IT guy IRL and I love KOF XIII, so I thought I'd try to help you devs in a way that most testers probably can't: by analyzing the network traffic during matches and incorporating the findings into the feedback. I'm no network expert, but I hope that what I know about the way internet communication - or network communication in general - works will still be helpful.

Before proceeding, be warned: things are going to get quite technical for the average reader, bring your geek glasses! B)

What did I actually do differently than other testers?

Basically, I started Wireshark, a graphical front-end and analyser to the wonderful packet capture tool, WinPcap (which is more or less the equivalent of tcpdump, the Linux program used for the same purposes). I then started the capture on the ethernet interface of my computer's network card, so that anything that is sent to higher OSI layers for processing is recorded. Then I went into online matchmaking mode inside the game and tried both creating lobbies and joining existing ones. Once my opponent or I quit the lobby, I stopped the capture and started looking at the results.

So how exactly can Wireshark's output be used to draw conclusions about the netcode?

There are probably hundreds of Howto-s on how to use Wireshark, but a quick overview is basically this: you get to see with nanosecond precision when a packet was sent or received by your NIC, who that packet came from or went to, and - probably most importantly - what that packet contained (if it wasn't sent through an encrypted channel). Since a fighting game netcode requires synchronisation, I assumed that if the data isn't encrypted, looking for a monotonously incrementing byte - or two hexadecimal digits, in Wireshark's representation of the data - will get me to where the communication keeps track of it.

As stated in the second post of this thread in the closed beta forums, KOF XIII SE uses UDP datagrams to communicate between players, once the matchmaking service concluded its job and peer ports are (auto)negotiated. Wireshark lets you easily strip away the unnecessary metadata from the UDP datagram to let you look at the payload, which is what the game's netcode actually uses. Once I have which byte or bytes represent the sequence number of the datagrams, I can use that to compare the timestamps of the two players' packets for this sequence number, and with a simple subtraction I get not only the latency, but also the offset in number of datagrams by which the communication is delayed.

Okay, so how does this "UDP payload" thingamajig look like and what can we learn from it?

This isn't an easy question to answer, since the game's netcode uses a number of differently structured UDP payloads during different events (in lobby, on the character select screen, during match, etc.), but since there are 60 input frames in KOF XIII per second, there ought to be 60 frames worth of input data sent between the players every second among the captured packets. This is too large a quantity to miss. Based on this, I found that the way one frame worth of inputs (and possibly other fight-related data such as meter totals) is transmitted looks something like this:

00:00:22:03:00:00:b3:03:00:00:00:00:00:00:01:00:00:00:01:00:00:00:01:00:00:00:01:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:0a:00:00:00

The first six bytes are more or less always the same, so we needn't bother with those. The 7th and 8th bytes are much more interesting. In the example above, b3:03 are actually the sequence number in hexadecimal representation. How do I know this? Well, the data for the next frames had b4:03, b5:03, etc. . At the start of the match it was 00:00, and after ff:00 came 00:01... pretty convincing.

The rest of the bytes appear to be the inputs the player is pressing/holding and other stuff we needn't concern ourselves about right now...except for the last 4 bytes. If present, these indicate that this was the first time the data for this input frame was transmitted. That makes these "initial transmission" UDP datagrams have 88 bytes of data. Now, if something goes wrong, the netcode retransmits the data of certain input frames. These are easy to notice in the packet capture's output, since it breaks the sequence and lacks these last 4 bytes, making the payload size 84 bytes.

What the game actually does with this retransmitted input frame data is not clear, but from the examples that are to follow, it will become apparent that these are a reliable indication of how laggy and unresponsive a match feels like online. Before I move on to the actual match data, one last thing worth investigating is what the packet data looks like when the words "Synchronizing" pop up mid-match.

Against one of my opponents from the US, I had a terrible connection, lots of dropped inputs and what felt like a huge, quarter-to-half-second delay on occasions. I looked at the approximate timestamp at which I saw the "Synchronizing" message appear, and saw that he was actually retransmitting the frame with the same sequence number over and over again, for like 150-200 attempts. What's worse, they had the extra 4 bytes at the end which apparently denote that data as "initial transmission"! If that was the case, no wonder the game was stumped. Then some differently structured UDP datagrams came to the rescue and fixed the connection to the slightly less horrible state it had before the desync.

So, how did the matches look like through the eyes of Wireshark?

Just for the record, I'm playing from Hungary, Europe.

Player #1
Player name: Lupin
Host: me
Country: US (specific location, anyone?)
Connection level: 1 bar
# of matches: 4
Netcode version: v4.1

Using the timestamp subtraction and sequence number offset methods described above, I calculated the latency and input frame delay at roughly 90-second intervals to get a picture of how these performance counters evolved throughout the match:

A few seconds after the start of the round:		131 ms		6 frames
90 seconds after the first sample:			169 ms		19 frames
90 seconds after the second sample:			171 ms		8 frames
90 seconds after the third sample:			178 ms		9 frames

Now, ~10 frames worth of delay is more or less the difference between you reacting to stuff in time or getting hit by them, so you can tell that this match was not heavy with reaction anti-air supers and HD combos and neither were the ones that followed. The more disturbing thing was that many inputs just weren't happening the way I had pressed them and it looked the same on Lupin's side as well, from what I could tell. These were observable by micro-stutters during movement. What seems to correlate with this is the fact that there were 6-10 retransmitted input frame data packets every second, and these were usually for a sequence number 10-20 frames in the past.

Player #2
Player name: Deals
Host: him
Country: Germany (specific location, anyone?)
Connection level: 3 bars
# of matches: 3
Netcode version: v4.1

A few seconds after the start of the round:		2 ms		0 frames
90 seconds after the first sample:			17 ms		1 frame
90 seconds after the second sample:			33 ms		2 frames
90 seconds after the third sample:			31 ms		2 frames

This match was incredibly smooth, there was barely any difference compared to playing offline, to be honest (then again, I don't usually do three-1-frame-link kill combos with Mature or double-HD-activation ones with Vice like Juicebox ;) ). I was able to do combos befitting my beginner skill level and react to full jumps with supers. The "Synchronizing" message only appeared at the character select screen, where it was not so intrusive as during matches with Lupin. What's more, there were hardly any input frame retransmissions in the packet capture data! Basically they were only to be found once every 2-3 seconds. There were no micro-stutters as far as I can recall either. Landing the Target combo -> cr.B link with Kensou came naturally unlike with the 1-bar matches. The packets were also much less likely to arrive in bursts, it was "one from my PC, one to my PC" 85% of the time, with mostly 1 or 2 frames of difference, according to the sequence numbers.

Whew! That took an age to write, and most people will probably just TL;DR past it, also, apologies for any typos or copy-paste errors I might not have spotted. Anyways, if any SNK dev took the time to read this: if you have further checks in mind that I can perform to help you, I remain at your disposal. Just keep in mind that working hours for Central Europe and Japan have little to no overlap, so real-time communication might be tough to organise. I really want to help the PC version have the best netcode possible, so I hope all this rambling has been of help and I'll gladly help investigate further aspects I might not have thought about.

RETSU GO, KOF XIII SE!

[LazieFred]

Nice analysis. Keep this coming!

[Reiki_Kito]

Holy crap, that was a large amount of data. This is impressive!

[bmurphy88]

Wow! So hopefully the SNK crew comes back to work in a few hours.

[Reiki_Kito]

Heya Sny83, how's it going? I was curious if you've noticed any drops on your end in connection or if you've anymore data collected over the last few days.

[Reiki_Kito]

Good games! Those were pretty good matches, I had a hard time dealing with your jumping, but I have no regrets when I lost. You played very well!

[sny]

Hey rocky, thanks for the matches! I actually communicated with Jace and sent him some of the data I collected for analysis. I recorded a ton more matches/packet data tonight as well (from our matches as well) and will do the analysis for them tomorrow when I get home for work. Fear not, I'm working behind the scenes, just like SNK =)

[Killey]

Haha...as an IT guy as well I was able to follow your explanation. While I use wireshark as well I don't have the experience nor patience to go through a wireshark packet capture to that degree but your break down of it makes sense.

[sny]

Sorry for not posting this earlier, but I had a very busy and exhausting Wednesday/Thursday so I had no time to look at the matches from the beginning of the week until now. So, without further ado, here goes:

Player #3
Player name: TarMa971 (hope I got the numbers right ^^)
Host: him
Country: France (specific location, anyone?)
Connection level: 2 bars
# of matches: 6
Netcode version: v4.1

A few seconds after the start of the round:		34 ms		2 frames
90 seconds after the first sample:			33 ms		2 frames
90 seconds after the second sample:			33 ms		2 frames
90 seconds after the third sample:			29 ms		2 frames
90 seconds after the fourth sample:			36 ms		2 frames
90 seconds after the fifth sample:			37 ms		2 frames

These were short matches, since my opponent was leaps and bounds beyond my skill level, but we played 6 to compensate for it, so I had a chance to look at whether there was any deterioration in the quality of network latency or increse of frame delay. The numbers speak for themselves: there's none to be seen and I felt none either during the matches. For me it felt like it was hard to react to hop-ins, but this is most likely because this is the first time I ever played against a Ralf... damn, that guy has a low hop arc! Very hard to react to, and TarMa could convert into massive damage each time. I could barely land a hit on him, but the netcode had nothing to do with it. He was only slightly less dominant with his other point characters, ultimately leaving me no chance to see whether I could perform combos or drive cancels. All in all, it was a good quality match, only slightly less impressive than against Deals from Germany.

Player #4
Player name: PBF
Host: him
Country: Stockholm, Sweden
Connection level: 2 bars
# of matches: 6
Netcode version: v4.1

A few seconds after the start of the round:		45 ms		3 frames
90 seconds after the first sample:			27 ms		2 frames
90 seconds after the second sample:			40 ms		2 frames
90 seconds after the third sample:			32 ms		2 frames
90 seconds after the fourth sample:			39 ms		3 frames
90 seconds after the fifth sample:			38 ms		2 frames

I played these matches after midnight following a long day at work, so I was half asleep by that time, but from what I can recall, it was basically as fluid as against TarMa earlier. PBF also knew his game very well, but I could react to his moves at least. I was dropping my combos a lot of the time, but in retrospect I'd give the benefit of the doubt to the netcode and attribute it to falling off the chair to sleep (and being on the 2P side, which I never really practiced :( ). As with TarMa before, no deterioration over time was observed, and no stuttering or disconnects occured during the matches.

Player #5
Player name: Tom Sawyer
Host: me
Country: France? (isn't he Morroccan? specific location, anyone?)
Connection level: 1 bar
# of matches: 3
Netcode version: v4.1

A few seconds after the start of the round:		67 ms		3 frames
90 seconds after the first sample:			64 ms		4 frames
90 seconds after the second sample:			66 ms		4 frames
90 seconds after the third sample:			49 ms		3 frames
90 seconds after the fourth sample:			104 ms		6 frames
90 seconds after the fifth sample:			62 ms		4 frames

This was the worst quality match out of all those I played against Europeans (unless Tom Sawyer really does live in Morrocco, and only his ISP is reported as being french), but it was still better than your average console connection. I could tell he had a hard time landing combos and mistimed some blocks as well, and there were slight stutters every now and then during matches. In two of our three matches, we had the "Synchronizing" message pop up mid-match (one such occurrence came shortly after the fifth sample you can see in the code block below, so the game probably forces resync after a certain frame delay threshold is reached). It would be really advantageous to know whether he had the required ports forwarded on his router at the time, I had the feeling he might not have done that, as other European opponents all had better connections.

Player #6
Player name: ? (forgot to write down :()
Host: me
Country: Saudi Arabia
Connection level: 1 bar
# of matches: 2
Netcode version: v4.1

A few seconds after the start of the round:		65 ms		5 frames
90 seconds after the first sample:			85 ms		5 frames
90 seconds after the second sample:			74 ms		5 frames
90 seconds after the third sample:			63 ms		4 frames
90 seconds after the fourth sample:			70 ms		4 frames
90 seconds after the fifth sample:			76 ms		5 frames

This felt pretty much like the previous matches vs. Tom Sawyer, but without any cases of resynchronization mid-match (as far as I can recall). There were occasional stutters or frame skips, and the delay could be felt, but it was passable for a 1-bar connection.

Player #7
Player name: forcedgyp
Host: me
Country: Belgium
Connection level: 2 bars (it jumped to 3 bars occasionally)
# of matches: 6-7
Netcode version: v4.1

A few seconds after the start of the round:		32 ms		3 frames
90 seconds after the first sample:			30 ms		2 frames
90 seconds after the second sample:			46 ms		3 frames
A few seconds after the start of set #2:		34 ms		2 frames
90 seconds after the fourth sample:			45 ms		3 frames
90 seconds after the fifth sample:			43 ms		3 frames

All in all a fluid match, no skipped frames and mid-match resyncs. Hit-confirming and anti-airing was hard but possible, and the quality was the same throughout the two sets of matches. My opponent left and rejoined the lobby after a match or two, but it had no noticable impact on further matches.

Player #8
Player name: Kogsy
Host: me
Country: Austria? (the DNS lookup yielded no ISP, so it's hard to be sure)
Connection level: 3 bars
# of matches: 1
Netcode version: v4.1

A few seconds after the start of the round:		25 ms		1 frame
90 seconds after the first sample:			44 ms		2 frames
90 seconds after the second sample:			41 ms		2 frames

It's such a shame he left so early, because the match we played was really smooth, lagless and responsive. He looked very new to the game so it made my job easier, but I could land combos and drive cancels without fear of dropping them due to delay.

Player #9
Player name: rockyminoma
Host: me
Country: San Francisco, USA
Connection level: 1 bar
# of matches: 8
Netcode version: v4.1

A few seconds after the start of the round:		92 ms		6 frames
90 seconds after the first sample:			103 ms		6 frames
90 seconds after the second sample:			57 ms		3 frames
A few seconds after the start of set #2:		118 ms		7 frames
90 seconds after the fourth sample:			111 ms		7 frames
90 seconds after the fifth sample:			107 ms		6 frames

Despite it being a connection from Hungary to the West Coast, this match was still much more playable than 2-3-bar matches on the PS3! It had delay, sure, but it was playable. Rocky did a fine job at hit-confirming in spite of the slight lag, I had some trouble but that's because of my lack of experience and practice in KOF. He played a solid keepaway/zoning team, but as opposed to similar situations on console I was able to get around his fireballs from time to time, so the matches had a good back-and-forth feel to them, with his experience and scary Terry putting him on top in the end. Very rarely there were frame skips, and pre-match synchronizations took a bit longer than versus my other opponents, but I wouldn't scoff at the quality of connections like this knowing it was between continents.

On friday and during the weekend I'll get some V6 testing done and repeat the packet capture/analysis I did for these. I hope to have it ready by Monday Japan time. ^^

Keep up the good work, SNK!

[RMS-099 Rick Dias]

sny83, I can only follow some of that information... but the parts I do understand suggest this should be pretty useful to SNK or give them some ideas on what to look into. Thank you for sharing it!

[sny]

Here's some feedback from the latest beta version. I'll keep adding to this post as I meet more players and go through the data.

NOTE: I changed the sampling strategy from taking snapshots every ~90 seconds to taking snapshots at roughly even intervals throughout length of the set of matches, to better highlight possible deterioration of netplay quality (if any exists).

Player #10
Player name: Rayden-sama
Host: me
Country: France
Connection level: 1 bar
# of matches: 7
Netcode version: v7

A few seconds after the start of the round:		63 ms		4 frames
Second sample:						80 ms		5 frames
Third sample:						86 ms		5 frames
Fourth sample:						67 ms		5 frames
Fifth sample:						60 ms		4 frames
Final sample:						84 ms		5 frames

These were playable matches but with a LOT of synchronization (15 seconds or more!) happening at the character select screen, order select screen and at the start of each round. It was hard to react to jumpins, but my opponent was more of a defensive player so he outlasted me long enough to punish my execution mistakes with his hard-hitting team. Strange thing is I couldn't find any deterioration which would explain the high number of synchronizations in the packet data, but there they were anyway.


Player #11
Player name: Plasma-funk
Host: me
Country: Hungary
Connection level: 3-4 bars
# of matches: 1
Netcode version: v7

A few seconds after the start of the round:		32 ms		2 frames
Second sample:						24 ms		1 frames
Third sample:						133 ms		3 frames
Fourth sample:						62 ms		2 frames
Fifth sample:						41 ms		3 frames
Final sample:						14 ms		1 frames

This was the only match in which I clearly faced someone with a PC that is affected by the 100% CPU usage "bug" of KOF XIII SE. I don't know the guy personally, but I know a friend of his and he told me that his PC meets the minimum requirements, but it's not much better than that and that they had horrendous video lag when they tried playing. Well, that was visible on my end too: the match felt like it was being fought underwater, tons of micro-stutters which made combos almost futile to try, although it was possible to react to jumpins. The game skipped frames left and right, which is probably because Plasma_funk's PC was being pushed to the limit. I saw packets sent and resent in batches of a dozen mid-matches, and outside of matches there were constant synchronisations during which I got like ~200 packets from him to get his client to where mine was in the sequence of actions.

Player #12
Player name: HIGH/(^_^)/\(^_^)\FIVE
Host: him
Country: Belgium
Connection level: 1 bars
# of matches: 6
Netcode version: v7

A few seconds after the start of the round:		76 ms		5 frames
Second sample:						23 ms		1 frames
Third sample:						34 ms		2 frames
Fourth sample:						59 ms		4 frames
Fifth sample:						28 ms		2 frames
Final sample:						46 ms		3 frames

Overall these matches were playable, little to no synchronization going on, the message only appeared at the character select screen once in a while and not for a long while. There were some occasions when I could've sworn my reaction anti-airs were in time but they weren't, but looking at how the frame delay fluctuated over the course of the matches, that could very well account for it. Even so, these were enjoyable matches and doing long combos and hit-confirms was very much possible (even if my opponent managed it far more consistently than I have ^^).

Player #13
Player name: Erik
Host: me
Country: Denmark
Connection level: 2 bars
# of matches: 1
Netcode version: v7

A few seconds after the start of the round:		28 ms		2 frames
Second sample:						42 ms		2 frames
Third sample:						14 ms		1 frames
Final sample:						38 ms		2 frames

The best quality match I had in v7 so far, this felt just like offline. I could pull off my bnb-s and do some longer Kensou corner juggles as well. No stuttering, no synchronization message, no deterioration. I wish every match went like this!

Player #14
Player name: Tom Sawyer (again!)
Host: me
Country: France
Connection level: 1 bars
# of matches: 1
Netcode version: v7

A few seconds after the start of the round:		45 ms		3 frames
Second sample:						57 ms		3 frames
Final sample:						50 ms		3 frames

This will be good for comparison purposes, since I've met him under v4.1 as well (see posts above this one). The connection was much better than the last time we played and it showed, since his middle character (Chin) pretty much demolished my last two characters. I couldn't even touch him (he's the first guy I met who uses the character as well). There was a small delay to be observed, but the stuttering from the previous encounter was gone and we could both react to each other's jumps fairly well. All in all, if this is indicative of how much of an improvement v7 is over v4.1, I'm pretty happy, since it turned an unplayable match to a playable one. I did get the "Synchronizing" message at the start of each round, but not during matches anymore.


Player #15
Player name: Aelsca_meyen
Host: me
Country: Canada
Connection level: 1 bars
# of matches: 10
Netcode version: v7

A few seconds after the start of the round:		156 ms		9 frames
Second sample:						132 ms		8 frames
Third sample:						139 ms		8 frames
Fourth sample:						125 ms		8 frames
Fifth sample:						134 ms		9 frames
Final sample:						143 ms		9 frames

Unfortunately this overseas match was unplayable from start to finish. While there was no stuttering or frequent synchronization, the delay was killing any attempt at prudent gameplay. Full jumps weren't possible to react to (let alone hops...) and even basic juggle combos were impossible to perform due to having to input the motions a quarter of a second sooner than what you were used to. This was more or less identical to how console netplay looks like for me currently vs. European opponents. All in all, not a pleasant experience, but at least it didn't deteriorate further into the set. We talked with Aelsca_Meyen during the matches and he pretty much had the same delay that I did. His ports are correctly set up and he's not suffering from any CPU issues either, AFAIK. A damn shame, because we appear to be close in KOF XIII experience, so practicing together would've been fun. :(

[youtube.com/@oyajigunsmith]

Hi Syn83, this is great stuff! FYI Tom Sawyer is Moroccan but lives in Paris.

[SNK.Prog_Jace]

Really appreciating all this effort Syn! Thanks again for feedback! Very interesting stuff.

Jace

[Vanscot]

Syn83: if you can walk me around gameshark, would you like a match where both players capture the traffic and compare results?
I'm from Chile, so expect a 1 bar fight but it could still provide some useful info.

[sny]

Guys, it's sny, not Syn. ;)

Vanscot: that could prove useful indeed, but I wouldn't recommend sharing captured data from your network traffic with anyone you don't trust, unless you know how to filter only the relevant packets. That said, if you're still interested, add me and we can discuss details. ^^

[Narisa Kanael]

For that matter, I would be happy to help you again, but for next time, If possible, can we use VPN? I'm having another problem with filtering off the relevant packet, so.

That delay suprised me a lot as well..