STEAM GROUP
Steam In-Home Streaming homestream
STEAM GROUP
Steam In-Home Streaming homestream
5,347
IN-GAME
32,415
ONLINE
Founded
November 7, 2013
 This topic has been pinned, so it's probably important
GeeEl Feb 12, 2014 @ 1:35am
Using the Steam In-Home streaming logs
I was looking through the streaming logs and (after initially being quite intimidated by the lack of line breaks) was thinking of asking if the metrics meant what I thought they meant.
It turns out that asking about that is 80% (edit: 47%) of the way to making a beginner's guide - so here's a draft which may be useful.

Please comment if I have anything wrong, or if you have suggestions on information to add (or know where the proper place to post a guide would be for IHS).

What to check out before digging into the logs
If you're after some quick information on performance or potential fixes have a look at the following pages (in fact look at them anyway, because they're very useful):
Diagnosing Streaming issues with the IHS logs
As mentioned in the pages above there's an easily accessible streaming performance display available while playing (by pressing GUIDE+Y or F6).

In addition there are logs stored on the host machine which give an overview of performance at the end of a streaming session.

These logs relate to the "Display" latency shown in the on-screen streaming statistics and network latency (in streaming_log.txt) along with some information on game latency (in the *Trace.txt files).

Looking at the steaming logs lets you:
  • see session overview stats
  • drill down to see how each step the game video takes towards rendering on the remote machine is performing.
  • compare performance of different settings (by comparing the overviews of different sessions)
  • possibly identify specific performance bottlenecks during the beta

Finding your streaming logs
If you're streaming from a Windows host you can find your general In-Home Streaming logs in the folders below (without the x86 bit if your machine is running a 32bit windows).
Directory of C:\Program Files (x86)\Steam\logs 12/02/2014 06:54 AM 1,700,905,420 StreamAudioTrace.txt 09/02/2014 09:34 AM 167,143 streaming_log.previous.txt 12/02/2014 06:59 AM 67,312 streaming_log.txt 12/02/2014 06:54 AM 867,258,814 StreamVideoTrace.txt
If using a Linux host (which I haven't yet) it should be trivial to find the streaming logs folder with a trusty find command:
find ~ -name "streaming_log.txt"
For SteamOS hosts the logs are found under
/home/steam/.local/share/Steam/logs /home/steam/.local/share/Steam/streaming

For MacOS:
Originally posted by ♥♥♥♥matic2000:
OS 10.6.8 Mac users should find their logs in
[user name]/Library/Application Support/Steam/logs

The two *Trace.txt files hold massive detail about audio and video at the frame by frame level. They are kept until cleared manually but only collect data while the streaming graph is displayed. A much more concise versions of them can be taken with the screenshot capture.

If you end up capturing a screenshot and detailed video/audio stats for part of a session they end up as zips in:
Directory of C:\Program Files (x86)\Steam\streaming 12/02/2014 06:41 AM 1,469,755 Dark_Souls-_Prepare_to_Die_Edition_(211420)_02-12-14_06-41-08.zip 12/02/2014 06:53 AM 4,337,432 Tomb_Raider_(203160)_02-12-14_06-53-41.zip

Looking inside one of the zips we can see they have a cut down chunk of the *Trace.txt files and a nice screenshot.
12/02/2014 06:41 AM 3,138,700 Screenshot.png 12/02/2014 06:41 AM 790,068 StreamAudioTrace.txt 12/02/2014 06:41 AM 218,953 StreamVideoTrace.txt

For the moment I'm just looking at the overview stats from the gaming session, which are found in the much lighter streaming_log.txt

Looking at session statistics with streaming_log.txt
For an example I'm going to have a look at DiRT 3, which runs pretty well on my system.

Note: The streaming_log.txt file uses UNIX style line breaks, which Notepad doesn't handle well. If viewing it on a Windows system use Wordpad/write (maybe even Word) to preserve line breaks

[2014-02-20 23:40:05] ===================================================================== Game: DiRT 3 (44320) [2014-02-20 23:40:05] Recording on device: Realtek HDMI Output (ATI HDMI Audio) [2014-02-20 23:40:05] Audio client mix format: [2014-02-20 23:40:05] format: 65534 [2014-02-20 23:40:05] channels: 2 [2014-02-20 23:40:05] samples/sec: 48000 [2014-02-20 23:40:05] bytes/sec: 384000 [2014-02-20 23:40:05] alignment: 8 [2014-02-20 23:40:05] bits/sample: 32 [2014-02-20 23:40:05] channel mask: 0x3 [2014-02-20 23:40:05] data format: {00000003-0000-0010-8000-00AA00389B71} [2014-02-20 23:40:05] Initializing audio with 2 channels and 48000 samples/sec [2014-02-20 23:40:10] Detected 4 logical processors, using 2 threads [2014-02-20 23:40:10] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Capture method set to Desktop BitBlt RGB [2014-02-20 23:40:31] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Capture method set to Game async D3D11 NV12

The header shows some basic details about the capture methods being used.
It should be mostly the same for every streaming session, but what does look interesting is the Capture method:
Capture method set to Desktop BitBlt RGB Capture method set to Game async D3D11 NV12

In this case D3 (aka annoyingly mixed-case DiRT 3) is using a newer DirectX 11 "asynchronous" capture method, meaning that it can do some extra steps in parallel.

Now we get to the performance information - the session statistics.

[2014-02-20 23:48:31] "SessionStats" { "TimeSubmitted" "1392904111" "ResolutionX" "1280" "ResolutionY" "720" "CaptureName" "Game async D3D11 NV12" "DecoderName" "VDPAU hardware decoding" "BandwidthLimit" "10000" "FramerateLimit" "50"
In this case, I'm running in 720p (ResolutionY) as my (recently upgraded) host machine has a CPU circa 2009 (an AMD Phenom II x4 945 - newer than I thought).
My SteamOS client is using VDPAU hardware decoding (DecoderName). According to wikipedia this is open source and courtesy of Nvidia - thanks Nvidia!

I've manually limited bandwidth to 10Mbps, and left framerate as Automatic - IHS has decided that a cap of 50 frames per second would be best.

The next chunk of SessionStats shows obvious problems that are causing poor performance:
"SlowSeconds" "0" "SlowGamePercent" "0" "SlowCapturePercent" "0" "SlowConvertPercent" "0" "SlowEncodePercent" "0" "SlowNetworkPercent" "0" "SlowDecodePercent" "0"
The Slow* chunks are pretty self exlanatory.
SlowGame - caused by the game itself running slow
SlowCapture - caused by copying the displayed frame into memory taking too long
SlowConvert - Erm... TO BE COMPLETED...
SlowEncode - caused by the CPU (or hardware encoder) taking a long time to encode a frame in to x264.
SlowNetwork - caused by network connection problems, or a lot of competing traffic on the network.
SlowDecode - caused by the client machine taking a long time to decode the frames.

Next we get onto the standard format for the performance statistcs, average (Avg) and standard deviation (StdDev) statistics for the streaming performance metrics.

From the Wikipedia Page on Standard deviation[en.wikipedia.org] (give them a donation or help fix a page - they love that):
In statistics and probability theory, the standard deviation ... shows how much variation or dispersion from the average exists. A low standard deviation indicates that the data points tend to be very close to the [average] a high standard deviation indicates that the data points are spread out over a large range of values.

For our purposes if the StdDev is small compared to Avg it means that there's generally consistent performance for this metric. If it's large it means that that metric could be causing spikey delays.

If the stat measures time it'll be in milliseconds.

"AvgClientBitrate" "63.820789337158203" "StdDevClientBitrate" "16.055004119873047" "AvgServerBitrate" "8809.0078125" "StdDevServerBitrate" "14545.39453125"
ServerBitrate - the video+audo bitrate that the x264 encoder streaming to the client in Kbit/s.
ClientBitrate - bitrate the client is sending data (input and data confirmation) back to the server in Kbit/s

"AvgLinkBandwidth" "61510.49609375" "AvgPingMS" "1.1842796802520752" "StdDevPingMS" "0.25608810782432556"
LinkBandwidth - An estimation of available network bandwidth in Kbit/s
PingMS - Network round-trip time between the host and remote client
"AvgCaptureMS" "5.708702564239502" "StdDevCaptureMS" "2.6058037281036377"
CaptureMS - How long it takes to make a copy of the current frame for encoding
"AvgConvertMS" "4.9717216491699219" "StdDevConvertMS" "1.6487746238708496"
ConvertMS - TO BE COMPLETED
"AvgEncodeMS" "12.187912940979004" "StdDevEncodeMS" "2.8964734077453613"
EncodeMS - Time taken to compress the frame for transmission to the client. This is one of the most expensive steps and is very CPU intensive (until hardware encoding is supported).
"AvgNetworkMS" "0.56417733430862427" "StdDevNetworkMS" "0.11537900567054749"
NetworkMS - Time taken transferring the encoded frame to the client over the network. In a good setup this should be negligible.
"AvgDecodeMS" "3.3226580619812012" "StdDevDecodeMS" "1.5122100114822388"
DecodeMS - Time taken by the client to decompress the video frame
"AvgDisplayMS" "4.4955034255981445" "StdDevDisplayMS" "4.8185925483703613"
DisplayMS - Time taken to display the frame on the TV/Monitor
"AvgFrameMS" "43.798892974853516" "StdDevFrameMS" "10.485637664794922"
FrameMS - Total time taken to stream the frame.
"AvgFPS" "46.833816528320312" "StdDevFPS" "7.0818371772766113" }
FPS - Frames per second

Looking at SteamVideoTrace.txt
As mentioned, SteamVideoTrace.txt in the logs folder is a massive file. The better way to go about looking at the frame by frame detail is to capture a screenshot zips (stored in the Steam/streaming directory). The capture of the data and screenshot is triggered in game with pressing F8 (keyboard) or GUIDE+X (controller) and can only be done while the performance graph is displayed.

It gives the same sort of detail as the performance log above, but with detail going down to individual frame by frame timings.

The below chunk is from streaming Tomb Raider (2013)
NETWORK: Ping: 1.19ms, Server: 9603 Kbit/s, Client: 64 Kbit/s, Link: 81516 Kbit/s, Packet loss: 0.00% Frame: 2582, 24573 bytes, k_EStreamFrameEventStart at 60887.93ms k_EStreamFrameEventCaptureBegin at 60908.10ms, delta: 20.17ms k_EStreamFrameEventCaptureEnd at 60915.61ms, delta: 7.51ms k_EStreamFrameEventConvertBegin at 60915.61ms, delta: 0.00ms k_EStreamFrameEventConvertEnd at 60915.61ms, delta: 0.00ms k_EStreamFrameEventEncodeBegin at 60915.61ms, delta: 0.00ms k_EStreamFrameEventEncodeEnd at 60926.22ms, delta: 10.62ms k_EStreamFrameEventSend at 60924.76ms, delta: -1.46ms k_EStreamFrameEventRecv at 60925.36ms, delta: 0.60ms k_EStreamFrameEventDecodeBegin at 60925.54ms, delta: 0.18ms k_EStreamFrameEventDecodeEnd at 60927.81ms, delta: 2.27ms k_EStreamFrameEventUploadBegin at 60927.99ms, delta: 0.18ms k_EStreamFrameEventUploadEnd at 60928.22ms, delta: 0.23ms k_EStreamFrameEventComplete at 60937.84ms, delta: 9.61ms total frame time: 49.91ms, result = k_EStreamFrameResultDisplayed
From this we can see exactly what's happening for individual frames.
Delta is math slang for change in value, so we can see that there's a 20.17ms gap between start of the frame and the start of encoding (where I guess the game is rendering the frame before it can be captured).

The StreamAudioTrace.txt has similar details for the sound data.

Some games, in this case Dark Souls (which uses Direct3D 9 and performs poorly for me over streaming), have an additional chunk between frames.
total frame time: 31.13ms, result = k_EStreamFrameResultDisplayed, latency = 88.71ms (I0.4/G57.2/D31.1) ---------> k_EStreamInputEventStart at 149000.78ms, delta: 0.00ms k_EStreamInputEventSend at 149000.78ms, delta: 0.00ms k_EStreamInputEventRecv at 149001.05ms, delta: 0.27ms k_EStreamInputEventHandled at 149006.61ms, delta: 5.55ms
I think this is covering an additional step required for frame capture under Direct3D 9 (if anyone comments, I'll add the details in here). It also shows the Input (I), Game (G) and Display (D) latency. Given the frame rate I was initially getting in Dark Souls I'd believe that the game sometimes took 57ms to render.

Some Troubleshooting pointers
Host performance
SlowGamePercent, SlowEncodePercent, CaptureMS, EncodeMS : These generally means your host CPU is struggling (or, for SlowGame, the game is just be a badly written, or use a display method Steam can't stream efficiently). I know you may hate the idea, but dropping back to 720p (or even 480p) inside the game is probably the easiest way to improve performance. Reducing resolution decreases what the machine has to render and reduces the data to capture and encode.
Hopefully hardware encoding will be supported soon (although hardware encode is reportedly not as flexible as software, so we'll have to see) and if that works out hopefully Slow Encode will become a thing of the past (on supported GPUs).

Network performance
SlowNetworkPercent, NetworkMS, PingMS: If these are high it's indicative of a poor network connection. This could be a a lot of competing traffic (data or, on wi-fi, interference) or a bad piece of network equipment (including flakey ethernet cables). If using wi-fi try moving closer to the wi-fi router and see if it helps. If on ethernet, try swapping cables/switch if you can (and if in a complex setup plug them into the same switch for a test).
Also check what the ServerBitrate is compared to LinkBandwidth. It may be that the bitrate in the In-Home Streaming settings needs to be manually set to a lower value.

Client performance
You're probably hard pressed to see problems at the client end (unless you're experimenting to see what the bare minimum specs for Streaming are), so unless something's gone seriously awry you probably don't need to worry about the client too much.
DecoderName: If this isn't some sort of hardware decoder your client is using CPU decoding (which isn't bad, but would restrict you to lower resolutions with a slow CPU. If your GPU/CPU has a hardware h264 decoder it may just be that it's not yet supported by the beta.
SlowDecodePercent, DecodeMS: This could mean a CPU bottleneck (if software decoding). In this case, it may be worthwhile experimenting with a lower bitrate (pushing more compression back to the host, and degrading video quality), reducing host resolution, or decreasing framerate. All of these should reduce the data being decoded. Note that decreasing the framerate if you've nailed the bitrate to a set value may actually increase latency (bigger individual frames) so experiment and see what helps.
DisplayMS: Bad video bandwidth/drivers (Any suggestions??)
Last edited by GeeEl; Jun 11, 2014 @ 1:15am
< >
Showing 1-15 of 69 comments
Rumbalotter Feb 12, 2014 @ 2:24am 
Awesome thanks @GeeEl for the effort!
GeeEl Feb 12, 2014 @ 4:47am 
Thanks. On the Dark Souls front I fiddled with DSFix 2.2 and managed to drop down to 1280x720, which got me up to a fairly consistent 24 FPS. It's still a crumby console port, but should be playable.
Gimpylung Feb 12, 2014 @ 5:04am 
Nice work.

Notepad++ (for Windows anyways) can handle the line breaks too
slouken Feb 12, 2014 @ 6:14am 
Looks great, thanks for the nice writeup! :)
ran346 Feb 12, 2014 @ 7:22am 
Thanks for putting this together. I will definitely look into this to iron out some of my streaming issues. Great work.
Hughmanhorn Feb 12, 2014 @ 7:48am 
Very informative. Thanks for putting in the time to write that. This information should go a long way to helping people understand what their performance metric means.

The following information looks odd to me too...


"AvgFrameMS" "62.845630645751953"
"StdDevFrameMS" "269.71429443359375"
FrameMS - Total time taken to stream the frame.

"I have no idea why the StdDevFrameMS is so large here - none of the other counters could account for it... Do I have this wrong?"


"AvgFPS" "46.290416717529297"<---does this mean you're performing below the average?
"StdDevFPS" "8.3101902008056641"
FPS - Frames per second
Last edited by Hughmanhorn; Feb 12, 2014 @ 8:44am
HeikkiH Feb 12, 2014 @ 8:13am 
Say10. StdDev is standard deviation - see for example http://en.wikipedia.org/wiki/Standard_deviation
Hughmanhorn Feb 12, 2014 @ 8:41am 
Originally posted by Heikki Hämäläinen:
Say10. StdDev is standard deviation - see for example http://en.wikipedia.org/wiki/Standard_deviation
I did an edit. I guess my question makes more sense if I point to the users fps, but what I don't understand about standard deviation in this case is where does the standard deviating average come from (is it 60fps?) and I assume I misspoke when I thought it could compare averages to other users.
HeikkiH Feb 12, 2014 @ 9:20am 
Average doesn´t mean that you are performing belowe average. It really means that this is typical fps which the game is showing. Deviation means how much it varies over time. Little bit simplified 2*deviation gives you area which covers 95 % of the values. In your case that means that most time your fps is between 30 - 62.
Last edited by HeikkiH; Feb 12, 2014 @ 9:21am
Hughmanhorn Feb 12, 2014 @ 9:47am 
So in GeeEl's case their frame rate was between 38 and 54fps with an average of 46?
GeeEl Feb 12, 2014 @ 12:19pm 
Originally posted by Say10:
So in GeeEl's case their frame rate was between 38 and 54fps with an average of 46?

The average is 46, and if the FPS values were in a "normal distribution" there'd be a nice 68-95-99.7 rule[en.wikipedia.org] about how much of the time the FPS value hovered around 46.

Unfortunately my one flirtation with statistics (which I failed due to not turning up very much...) was 21 years back now. A shame really, because (having looked at the text book later) it's quite an interesting field.

Anyone recall if we can say what percentage falls within one SD for non-normal (or even aproximations of normal) distributions?

We may only be able to say that "high SD = scattered results".
Hughmanhorn Feb 12, 2014 @ 1:04pm 
I fully understand what you're saying. What I failed to understand in your one post was that you were using 2 standard deviations (I should have realized when you said 95%) in your example and it confused me. Thanks for taking the time to sort me out. I learned something today!
SHOWT1M3 Feb 12, 2014 @ 7:10pm 
Can you add into the opening post how to read the graph, I have heard so many different theories on what the blue and red lines mean. We love Valve to properly verify.
GeeEl Feb 12, 2014 @ 11:50pm 
DaveW_uk did a nice write up: http://steamcommunity.com/groups/homestream/discussions/0/540732889636623310/
I believe that slouken's ^^ translates to "read above, good information".

I'll pop a link near the top of this one when I'm at a real keyboard.
...
Added links to the Davew_uk's write up, and the streaming support page (which I always seem to have trouble finding).
Last edited by GeeEl; Feb 13, 2014 @ 1:27am
leovic Feb 15, 2014 @ 5:23pm 
I have a question about the logs. Does the steam client collect the logs? I feel I've done a good amount of testing already. But I haven't experience any unique problem that the forums hasn't already mentioned. I know I could just go to the forums and suggest a feature or a fix. And I will do that. I'm just curious if Steam is automatically collecting the raw information right from the beta testers.
< >
Showing 1-15 of 69 comments
Per page: 15 30 50