[Bug] Exclusive mode can cause wallpaperRegisterAudioListener to return empty arrays

[sev]

I've noticed that audio input to Wallpaper Engine from an endpoint making use of exclusive mode can cause wallpaperRegisterAudioListener to return an array with all zero values if the sound gets too loud. Normally the Windows mixer clamps samples when it mixes/renders them, and exclusive mode bypasses this flow entirely; these values likely exceed whatever Wallpaper Engine expects and throws some internal exception. I'm no expert on Windows audio/Quartz or Wallpaper Engine so this is a high level assumption of how this problem might be occuring.

My specific use-case is with foobar2000 with the foo_out_wasapi plugin and selecting the same output foobar2000 is utilizing. However, testing other solutions shows it likely that any exclusive mode output can be capable of causing this; I tested WASAPI and ASIO. Possible reproduction cases:

• Use a hardware audio device with a special driver that allows for "bit-perfect" or "raw" output using exclusive mode and make loud noises; many higher end mics and sound cards commonly have this functionality
  
• Use an audio passthrough program like Virtual Audio Cable or VB-Cable and push loud audio to an output with WASAPI etc., such as in my use-case

Logs don't show anything useful and diagnostics did not find a conflict as expected. If more information is needed I can do more digging if I have time.

[Biohazard]

I'm not completely sure I'm understanding correctly what you're doing, but if you're letting another application 'exclusively' take control of an audio device, then Wallpaper Engine - or anything else - can't use that device for output or loopback input to begin with.

[sev]

It does work just fine, until an audio track that peaks starts to play. Exclusive mode only allows one application to use an output or input endpoint at a time for writing or reading, respectively. However Wallpaper Engine gets this information does allow it to continue working, but it seems unable to handle volume levels outside of the expected bounds, or so I assume. You can still read from an output even in exclusive mode if I remember correctly, if that's what it is doing.

[Biohazard]

I'm pretty sure that at some point the exclusive WASAPI plugin in foobar prevented the whole loopback device from working and it didn't even initialize. Perhaps something changed since then, I would have to try it again and see what happens.

The great thing about WASAPI loopback is that I don't have to worry about wildly vendor dependent formats or normalization like it happens with other vendor specific recording devices. It sounds like that won't be true anymore with exclusive mode which could be a problem if a mere volume scale doesn't suffice (there is already an option for this in the settings, but if the values already became invalid then they're stuck until you restart the program, yeah).

[sev]

Originally posted by Biohazard:

It sounds like that won't be true anymore with exclusive mode which could be a problem if a mere volume scale doesn't suffice (there is already an option for this in the settings, but if the values already became invalid then they're stuck until you restart the program, yeah).

I didn't try fiddling with the volume. I will do that and report back.

[sev]

Adjusting volume does scale the values of the array, yes, but it does not affect what happens when audio is clipping.
If I lower foobar2000's output to the point that audio is not clipping, I continue to get populated arrays. Whenever it clips though, the array is zeroed.
The highest peak I saw was 13.750563621520996 17.653276443481445 328.1449279785156 633.3804931640625 920.7395629882812 with the equalizer pumped to +32dB and -6dB limiter on; I have no idea on what scale these numbers are, so take that as you will. Is it logarithmic?

[Biohazard]

I'm afraid I tried this again and it still does not work. The loopback device cannot be initialized with foobar2000 + exclusive mode WASAPI plugin running and some Windows core module even prints out some debug text to the console when I debug this.

I don't know why it works for you, but so far I have tested this on two different dedicated sound cards and onboard sound and none of them allow the loopback device to start up under these circumstances.

The error code Windows gives is 0x8889000a - AUDCLNT_E_DEVICE_IN_USE:
"The endpoint device is already in use. Either the device is being used in exclusive mode, or the device is being used in shared mode and the caller asked to use the device in exclusive mode."

see here: https://docs.microsoft.com/en-us/windows/win32/api/audioclient/nf-audioclient-iaudioclient-initialize

I think the oddity is that it initializes on your system while it's not supposed to even work that far.

[sev]

Sorry, I thought it was using the VAC playback device, but it was actually using the recording device, which does work in exclusive mode (or WASAPI doesn't need exclusive access for recording devices). I guess you can no longer read from playback devices if exclusive mode is used, but I swear you could in an earlier version of Windows... Or I could be misremembering. All my operating systems have been strangely customized in some way so it could have been something I did.

So this is a pretty far out edge case, then. MME seems to clamp values to an expected range, and the majority of users I imagine are not going to be using any exclusive mode audio, and even if they do, once they see it causes problems with Wallpaper Engine they'll probably stop using it.

This problem can realistically only occur using exclusive mode to bypass the clamping, and then having that audio directly available on another endpoint (a la VAC/VB-Cable); or otherwise get out of bounds data from an endpoint.

As mitigation in the mean time I can just lower foobar2000 output so things don't clip, and adjust my presets down the chain accordingly.

[Biohazard]

Is there still a way for me to replicate this without a certain setup of virtual devices?

In general there is no fixed limit on the incoming values and no clipping is done. When using vendor specific input devices, the values also have an unknown range since there is no general specification for that. So I'm not able to easily see the issue just yet.

[sev]

I have been messing around with it, but I haven't been able to see any particular occurrence of it within the boundaries of normal Windows usage. If I had Stereo Mix or such functionality I could try it with that but I have VAC specifically because I don't.

Is there any way I could debug this inside Wallpaper Engine, or with a specific build? Alternatively I can provide the means to replicate my setup.

[Biohazard]

There is one thing I'm seeing that I could try. Please switch to the beta here and wait for the next build which will include that change: http://steamcommunity.com/app/431960/discussions/2/350544272219415004/

I'm currently very busy with some other tasks that have a higher priority. So I really can't tell when I will be able to update the beta, sorry, it will likely take a couple of days at least or even a few weeks.

Originally posted by sleepy sev:

I have been messing around with it, but I haven't been able to see any particular occurrence of it within the boundaries of normal Windows usage. If I had Stereo Mix or such functionality I could try it with that but I have VAC specifically because I don't.

I have a Stereo Mix input (for testing purposes) and it works fine. I'm afraid you are the first one to have this issue.

Originally posted by sleepy sev:

Is there any way I could debug this inside Wallpaper Engine, or with a specific build? Alternatively I can provide the means to replicate my setup.

The only way I can afford looking into this currently is if there is a way to replicate this through WASAPI/Windows functionality. If it absolutely requires third party software then I would have to note this down for later because it will take a lot more effort to properly understand and it doesn't affecting the majority - but a 'fix' could change the behavior for a lot of people if I'm not careful.

I'm still not able to find any information on why this even works and what I should expect on my end in this situation. I have a strong feeling that the device isn't operating according to the standard because it should reject the request from Wallpaper Engine since it's already in exclusive mode. The code in Wallpaper Engine explicitly requests a shared mode connection too (since that's the only thing that makes sense there).

Perhaps the change I will push helps already then none of that should matter anymore, this change shouldn't affect normal WASAPI cases negatively.

[sev]

I understand.

I have opted in. Let me know when you push it and I will do more testing. Sadly I know nothing of Windows kernel/driver debugging so I can't provide any more information. Thank you for the hard work.

I do have a feeling that this may be related to VAC though because any device or software I use to grab the VAC input with WASAPI does not block other programs from reading from it, but in the VAC control panel it only shows one playback/render stream. To be clear, foobar2000 -> foo_out_wasapi -> VAC line output -> VAC processing -> VAC recording line input -> any program that reads from input using WASAPI. If it matters I am running foobar through VAC to Thimeo Stereo Tool, and then running that using kernel streaming to my DAC. I get the best sound quality this way.

[Tim]

Hey, could you check if this issue persists on the latest beta?

[sev]

Nope, still occurs.
Here's a video demonstrating it: https://anonfile.com/5eG0Peheo1/2020-03-16_18-13-32_mkv
As soon as it peaks every value in the array goes to 0. I'm not worried about peaking since I have a program after it that declips and normalizes the sound.Normally it won't go to flat 0 while playing a song... you can tell when DMIN (current min seen value from the array's values) hits 0 that something went wrong.