SteamVR

SteamVR

MisutaaAsriel Apr 14, 2024 @ 11:25pm
DRX, DDec Rhythmic Lag Spikes After Updates [ Steam Link ]
Issue


  • Previously, performance was phenomenal and without issue. Stuttering appeared within the last week of use, after a 2 week period of non-use, and various software and system updates.

System Information


  • Windows 11 build 22631.3296 (23H2)
  • Realtek Gaming Gbe Family Ethernet Controller
  • Realtek RTL8822CE 802.11ac PCIe Wireless Adapter
  • 802.11a 5Ghz 40Mhz Direct Wireless Network
  • Internet Hardwired
  • Microsoft QoS Policy for SteamVR set to Highest Priority, by Port & Executable
  • Meta Quest 2
  • Latest Steam Link build (as reported by Meta Store)
  • Latest SteamVR (as reported by Steam)
  • QoS detection disabled in Steam Link (Network Incompatibility)
Last edited by MisutaaAsriel; Apr 14, 2024 @ 11:29pm
< >
Showing 1-13 of 13 comments
charlesl Apr 15, 2024 @ 4:22pm 
Because the spikes are on "DRX" and also you can see the ping dots falling back slowly, it does appear to be a networking issue.

There are a lot of possibilities, but you can test this independently by using a tool like cnping https://github.com/cntools/cnping/releases/tag/1.0.0 to see if the ping also remains erratic when not streaming at all, or under different conditions.

Additionally, try to toggle the wifi off and back on in headset, going into/out of standby does not trigger the driver reload, but I am curious if that has any impact.

If you do not see spikes in cnping, when you do see spikes in the Steam Link detailed debug graph, please let us know!

There is another user who is experiencing something else like this who is also using a local USB streaming device.
MisutaaAsriel Apr 16, 2024 @ 11:52am 
Originally posted by charlesl:
There are a lot of possibilities, but you can test this independently by using a tool like cnping https://github.com/cntools/cnping/releases/tag/1.0.0 to see if the ping also remains erratic when not streaming at all, or under different conditions.

Test Results


  • Without load, Ping to Quest, 0.02s ➜[ibb.co] — As one can see, whilst there are "spikes", in a rhythmic-ish fashion, they aren't terribly severe, they are not quite as consistent, and the average results are also skewed by an initial large spike at the start of the test; I have seen the average drop down to 7ms. (It seems the Quest drops the first few pings?)
  • Without load, Ping to Quest, 0.05s ➜[ibb.co] — Upping the ping rate to 0.05s shows a different story, however. The ping spikes are more frequent, more rhythmic, and almost like a "heartbeat", occurring in regular intervals. There is also slight loss. It seems to remain consistently around a stable 15ms avg. albeit, in extended testing.
  • With load, Steam Link, Ping to Quest, 0.02s ➜[ibb.co] — Here we see heavy degradation, with the average ping increasing to 26ms, a historical max around 300ms that is not accounting for the initial "spike" noted above (it was lower before the spikes during the pictured test), and I have seen the average rise upwards towards 70ms. The numbers are erratic and the spikes grow more frequent and more severe.

Conclusion: It would seem that whilst there is some spikes in ping on a semi-regular basis, they are nominal when not under load, and using the default settings. When increasing the time between pings, there is slight degradation, and the "rhythmic" aspect appears, but the ping still hovers around a solid 15ms. It is not until Steam Link initializes that the numbers become severe, with spikes growing in both frequency and intensity.

Small Update


Unsure if related. but I used Microsoft's Network Monitor, ran the ping, and filtered it out in the monitor results, and I noticed that during the ping spikes, the Quest unit appears to be "phoning home" to a server owned/operated by Facebook/Meta. Specifically, it contacts 31.13.70.51 and other similar IPs (plus an array of oculus URLs). A lot. Is it possible a software update from the Quest is causing it to spam "home" in such a way that it degrades the connection? Or would this be a red herring?

Small Update #2


Reran the test in network monitor, but with Steam Link, and the RTP & UDP communications filtered out. I can confirm now that every time a lag spike happens, the Quest sends and receives a burst of communications with Facebook/Meta/Oculus servers. These bursts directly correlate to lag spikes in Steam Link. Unsure if side effect of greater issue, as correlation does not equal causation, or if Quest software is doing something it shouldn't, or if a combination of factors.
Last edited by MisutaaAsriel; Apr 16, 2024 @ 12:32pm
charlesl Apr 16, 2024 @ 5:27pm 
Let me start by saying: Impressive detective work!!! I will caveat some notes regarding your tests below.

I am really surprised by this - and that other users don't seem to be seeing similar issues, but would like to learn more about it. Sadly, I don't have insight into Meta's internal workings. I would be curious what you find.

Re: Test Results:

1. Those spikes are VERY severe! They are over 150ms (more than ten frames!) but, this is likely an artifact of your headset having the network screen up. If you close your network settings in the headset, they should not appear. My expectation is that they are actually network search events.

2. Your third image does give away quite some secrets. In that ping test, your spikes aren't spikes and more mountains, this happens when there is either network contention or too much bandwidth used for a specific link. I am curious how much bandwidth you were using when you hit them.

In general, the network profile from your third test, and your first message are inconsistent. As in, they are likely caused by different base reasons. Your first post shows sporadic peaks, and low bandwidth usage. Whereas your third test shows consistently queued packets.

Just pointing out your conclusion is the opposite of the take away that should have come from the tests. I.e. don't look at average or min, only look at max, and the period.
MisutaaAsriel Apr 17, 2024 @ 9:37am 
Originally posted by charlesl:
1. Those spikes are VERY severe! They are over 150ms (more than ten frames!) but, this is likely an artifact of your headset having the network screen up. If you close your network settings in the headset, they should not appear. My expectation is that they are actually network search events.

Perhaps I should have been more verbose, because the network screen wasn't up when these tests were taken! So if they are network search events, it's occurring even when the menu is close. It may or may not be possible in my case to mitigate this using ADB (I have a developer unlocked Quest), but this is suboptimal if so, as this means anyone else who might experience my issues in the future would also need to do so, and this does not solve the core issue.

To clarify, when tests 1 & 2 were taken, the headset was in the Quest home, with no apps open, and no menu displayed. I did reset the network before running these tests. But all three tests were done back to back as a set.

Originally posted by charlesl:
2. Your third image does give away quite some secrets. In that ping test, your spikes aren't spikes and more mountains, this happens when there is either network contention or too much bandwidth used for a specific link. I am curious how much bandwidth you were using when you hit them.

I would be curious too, though I'm not sure how to measure that, amusingly enough. I know the link speed is 866Mbps, but that's the theoretical max speed capable based on the connection, if I'm not mistaken. The bandwidth usage I imagine is much lower.



UPDATE: Whilst it's not reliable, vrserver.exe uses about 3.2MB/sec, or approx. 26Mbps, outgoing, and 340KB/s receiving, or approx 2.7Mbps, per Resource Monitor. I cannot find anything else in resource monitor that is using the connection, which suggests to me that the source of these spikes may be Quest-side? That is, if the only executable accessing the Quest's network is, in fact, Steam Link, then any lag must be attributable to the Quest itself. Is that line of thinking sound?



The Quest is also the only device on the network, not counting the host machine, and the network itself is limited to 1 singular device at a time (again not counting the host), so if there's any contention, it's not with other devices on the network (as there are none, in this regard).

Originally posted by charlesl:
In general, the network profile from your third test, and your first message are inconsistent. As in, they are likely caused by different base reasons. Your first post shows sporadic peaks, and low bandwidth usage. Whereas your third test shows consistently queued packets.

This is strange, and perhaps an inconsistency between cnping & Steam Link. Or, perhaps, if I had captured more of the debug window it would have given a better picture, as the peaks, whilst they may appear sporadic, were mostly consistent, even in the debug window, and the Steam Link debug window output is consistent between sessions as it was during Test #3.

For reference sake, I have edited the original debug window screenshot to outline the actual peak "sets" as they occur. ➜[ibb.co] Occasional sporadic peaks beyond this may occur, but what was highlighted has been consistent.

Just pointing out your conclusion is the opposite of the take away that should have come from the tests. I.e. don't look at average or min, only look at max, and the period.

I was factoring in average & min for the period mostly for the sake of ruling out a "generic" bad connection issue; I.E. if the connection were bad, the ping would be consistently bad as well, rather than these strange rhythmic "bursts", which suggest something with more curiosity and nuance behind it, which may be resolvable.

Update #2


Whilst researching my issue, I came across a similar issue seemingly plaguing the Quest 3. More specifically, the Quest 3, on v63, experiences network lag when the headset is in motion. ➜[communityforums.atmeta.com] Whilst I am on v64, not v63, and using a Quest 2, not a Quest 3, I can say that with some additional testing, the spikes do appear to be more frequent with movement. Furthermore, the issue linked is labeled as "Not Resolved" despite being marked solved.

Likewise, v64 was pushed out April 8th, which is within the 2 week window in which I was not regularly using the headset. I am beginning to think the most recent update may have caused issues with the Quest 2's networking hardware, or is otherwise introducing some new sort of load which the network is not fit to handle. Am I wrong to think this?
Last edited by MisutaaAsriel; Apr 17, 2024 @ 12:24pm
charlesl Apr 17, 2024 @ 7:27pm 
Your highlighting of the peaks does concur with my expectation for where they were happening. With cnping, it's easier to see rhythm of lag spikes, because it operates at a fixed speed, whereas with Steam Link, it will hold off frames if it knows they could not be displayed (like if there are still pending frames that would need to be displayed first).

There might be a bit of another answer here, re motion and lag spikes. Wifi has extreme locality to its quality. Even over small areas, you can find that certain positions and orientations of devices can have an order of magnitude different max throughput than positions and orientations even an inch or two away. So it is possible you are moving through the space causing different behavior.

*** What specific dongle are you using? Do you mind finding it's USB VID/PID in device manager so I could investigate that specific dongle? ***

This is generally only exhibited on older wifi hardware because newer wifi hardware can negotiate much better than older hardware can, though.

One of the thing that has me concerned/interested is what you found regarding the headset connecting to facebook's servers. My intuition would suggest that that only happens on the home screen though. Just something interesting to look into.

Thank you for pulling all these threads.
MisutaaAsriel May 3, 2024 @ 1:12pm 
Apologies for the delay, but I've done some testing which both further cements that it may be an issue with the Quest in my mind, and is rather curious in its behavior.

Originally posted by charlesl:
What specific dongle are you using? Do you mind finding it's USB VID/PID in device manager so I could investigate that specific dongle?

This is generally only exhibited on older wifi hardware because newer wifi hardware can negotiate much better than older hardware can, though.

I'm using M.2 PCIe adapters connected to the motherboard, not USB. As mentioned prior, it was an RTL8822CE, with 802.11ac (and configured in settings to only use 802.11ac 5GHz, with its b/g 2.4GHz disabled.

I have however switched to an Intel AX200 Wi-Fi 6 802.11ax card with external antennas. I did so both for its Wi-Fi 6 capabilities and its external antennas. This card is also considerably more configurable than the previous Realtek device, and allowed me to engage in some interesting experiments. To humor you, its Hardware ID is "VEN_8086&DEV_2723&CC_0280"

For the testing, this new card specifically allows me to lock the 5GHz network to a specific specification, rather than use the highest available. As such, I tested performance and checked the ping on 802.11n (Wi-Fi 4), 802.11ac (Wi-Fi 5), and 802.11x (Wi-Fi 6), with interesting results:

802.11ax (Wi-Fi 6)
  • Worst performance; with frequent image degradation and freezing.
  • Max ping frequently around 100+ ms
  • Rhythmic ping spikes
  • 1200Mbps Bidirectional Link Speed

802.11ac (Wi-Fi 5)
  • Bad performance; Comparable to the previous Realtek card
  • Max ping frequently around 70+ ms
  • Rhythmic ping spikes
  • 866Mbps Bidirectional Link Speed

802.11n (Wi-Fi 4)
  • Best performance; Almost no issues.
  • Max ping frequently around sub-40 ms
  • No rhythmic ping spikes, but observed new behavior.
  • 144Mbps Bidirectional Link Speed

In regards to the new behavior with Wi-Fi 4...
Unlike Wi-Fi 5 & 6, Wi-Fi 4 does not exhibit the rhythmic lag spikes. However, in its place, periodically there is a 3 second period of constant spikeage into the 60+ ms territory followed by a drop back down to its regular levels (Image Linked from a Severe Period[ibb.co]). During this period, low ping rates are observed, creating "mountains" in cnping. It is intriguing to me, as it makes me wonder if the Quest is changing its behavior due to the lower bandwidth connection.

I will be doing further testing.

Originally posted by charlesl:
One of the thing that has me concerned/interested is what you found regarding the headset connecting to facebook's servers. My intuition would suggest that that only happens on the home screen though. Just something interesting to look into.

You would think. But no, it did it also whilst in a Steam VR environment.

Update: For information's sake, here are some of the domains/IPs it contacts, coinciding with ping spikes:
Whilst it was hard to keep track (MS NetMon was having trouble parsing as fast as it monitored), it did appear that during spike periods on 802.11n, connection attempts to FB servers increased, judging by logged periods where connection attempts increased by a noticeable degree in a periodic fashion.
Last edited by MisutaaAsriel; May 3, 2024 @ 3:32pm
charlesl May 3, 2024 @ 5:37pm 
Thank you so much for all this information. I have shared some of your findings with other engineers here. Thank you for doing all this work and documenting it so completely.

I am shocked by your tests re: Wifi 5/Wifi 6, since it does not line up with my experiences here. It makes me wonder how different this would be in your setup with a router.

Also, I wonder if USB dongles would provide different behavior.

Anyway, if someone else stumbles across this thread, I hope they can continue to provide to the body of knowledge on this topic.
MisutaaAsriel May 3, 2024 @ 5:59pm 
Originally posted by charlesl:
I am shocked by your tests re: Wifi 5/Wifi 6, since it does not line up with my experiences here. It makes me wonder how different this would be in your setup with a router.

Just to remark: I was too, haha. Like, you would think the connection would improve with Wi-Fi 5, and further with Wi-Fi 6.

But as mentioned previously, signs now point to an issue with v64 of the Horizon OS which causes issues with 802.11ax networks, and in rarer cases, 802.11ac networks as well, based on the very large thread over on the Meta community forums.

I'm one of the seemingly first to experience this with a Quest 2 however (the thread focuses on Quest 3), but it would seem otherwise there's some strange bug effecting select units with recent software updates, going as far back as v63.

Seeing as Meta doesn't recommend using 802.11n (Wi-Fi 4) to begin with for such use cases as AirLink and kin, it does not surprise me that it seems the least affected by the bug, honestly. If any changes were made to their networking stack on the Quest side, it'd likely focus on 802.11ac/ax connections.

If other users experience these issues in the future, this thread is a good place to point them: https://communityforums.atmeta.com/t5/Get-Help/Not-Resolved-Known-Issue-Quest-PCVR-streaming-micro-lags-and/m-p/1158799#M279266


Update
In at-length testing, I went back to the Wi-Fi 6. The Wi-Fi 4 did have less periodic ping spikes, but in actual use the 3-second-long spikes were considerably worse than just occasional albeit regular dropped frames.
Last edited by MisutaaAsriel; May 12, 2024 @ 1:25pm
Ravensof Nov 5, 2024 @ 11:12am 
thanks for researching, it it helped find out the propblem (in my case).
i changed wifi 6 bandwith from 20/40/80/160 mhz to 20/40/80 mhz and frame drops disappeared
schmitzek Dec 8, 2024 @ 9:18am 
Hello everyone,
charlesl wrote: "I am really surprised by this - and that other users don't seem to be seeing similar issues, but would like to learn more about it."

I have had exactly the same problem for almost a year with the Quest 2. Periodic DRX spikes. I used Virtual Desktop before. At first I thought it was a problem with the new RX7900XTX graphics card, as there are fewer frame drops with HVEC codes.
But with Steam Link and this great debug tool I was able to narrow down the problem further.
I also think that the problem came with an update to the Quest 2 firmware. It may also depend on the hardware revision, as I also got a refurbished device before. Before that it worked perfectly for 2 years with an RTX2080.

I use a TPLink Archer AX10 dedicated router connected to the Realtek PCIe Gbit onboard Ethernet controller. Only for VR streaming. Windows 10.
I'm glad I found this post. I have now tried the following in the router settings:
- Bandwidth set from 20/40/80MHz to 20/40 or 20Mhz --> No success
- Encryption changed --> No success.
- Mode set from ax to a/n/ac mixed --> No success.

Best regards
I am experiencing constant, extreme hitching labelled as DRX in the debug graph.
Happy to help troubleshoot by providing any useful information.
Just chiming in to say I am experiencing the same issue.
However, my network setup is a bit weirder:

Host PC for steam VR -> router (as wired AP) -> router --5Ghz Wifi 5 connection--> Oculus Quest 3S


Worth noting that the issue does not appear in the setup with only one router:

Host PC for steam VR -> router --5Ghz Wifi 5 connection--> Oculus Quest 3S

Edit: nevermind. It seems the issue appears in both setups. Particularly if I move the headset.
Last edited by Warhatch technologies; Mar 1 @ 9:56am
I have solved the issue for myself. It was caused by a "wireless bridge" being on the same network.
< >
Showing 1-13 of 13 comments
Per page: 1530 50