MAJOR memory leak on Windows 10 Pro x64 [Killer Network Issue]

[Sebastian Nielsen]

**********************************************************************
UPDATE

The issue turned out to be the Killer network card. Download the base drivers (without killer features) from their website, and then spend some countless hours of uninstalling the Killer Network Manager (that was basically everywhere).

Also, set:
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Ndu\Start to "4".

After that, reboot computer, issue solved.

Have now verifyed and I see no larger usage than 2.5 GB, even when downloading about 150GB games.
**********************************************************************


ISSUE
Steam does not release back memory to the operating system after having written data downloaded while installing game data.

Im not entirely sure if this is a issue just in beta client, or if this is a issue that is also present in stable. Regardless, the issue should be fixed anyways.

A second issue, related to the first, is that Steam does not release memory allocated when quitting. This is a pretty severe issue, since this means that this memory will never be reused again, since the new Steam process will have a another process ID and no longer access the claimed memory.
OS
Windows 10 Pro x64 (with 32GB ram)

STEPS
Have a large game library with large parts not installed
Download every game in your library.
Have resource viewer up and notice how Steam just eats and eats more and more memory. However, it does not actively use the memory that it has claimed, so it will still look like Steam is using 85MB of memory.
Quitting steam also wont release the claimed memory.

RESULTS
Steam just eats and eats more memory. Regardless of how much memory you have in your computer, Windows will soon complain that theres not enough RAM on computer, especially if you have your paging file turned off.

EXPECTED
Steam should release back memory to the operating system after having written all game data to disk.
If this is not possible due to technical constraints:
If theres more games to download in queue, it could atlease reuse the RAM memory it did claim for the previous game to download the next game, so the maximum memory leak is contained to the size of the largest game in queue.

[Balderick]

I have noticed queued downloads have been getting scheduled to a later time rather than simply queued. This is likely because Valve are aware of the issue you mention and have applied that as a workaround for now.

My rig crashed earlier today with no obvious warning and no blue screen just a sudden hard power off when two fairly large downloads were queued. i wonder ...

[Sebastian Nielsen]

I have noticed downloads are scheduled for a later time, but only when the harddrive is slow, eg if I do downloads on a slow rotational disk, during high concurrent disk activity, or a networked disk during a network congestion.

Eg, it will try to download the queued download, notice its kinda slow to write to the disk, and after a few hundred MB, it just reschedule the download 10 minutes in the future.

Eg, when the green line is flat at the bottom for a while, Steam thinks "eeeh, computer is busy right now, lets wait and see if it clears up". Then it reschedule the download to optimize it.

------------------------------------------------------------------------------------------------------------------

HOWEVER:

This computer, my home computer Im sitting at now, however, is using a Intel 750 SSD 1.2 TB NVMe. That disk is BLAZINGLY fast, and I have never seen any downloads on this computer inside steam, to be scheduled for a later time.

Specs of this computer is:
Intel 5930k
GTX 980 Ti
Intel 750 1.2TB SSD NVMe
Gigabyte GA-X99M GAMING 5
Corsair Dominator Platonum 4x8GB DDR4 2400MHz
AX1200i

So I don't think the rescheduling thing is a fix for the memory leak issue, rather the rescheduling thing is to cover up if there's severe disk load or disk slowdown on the computer, since this rescheduling ONLY happens if the green line has been flat for a while.

I also think this rescheduling during slow disk activity is done to prevent severe fragmentation, because if 2 processes are interchangingly writing to the disk, it will basically lead to 100% fragmentation of both the file the first process is writing, but also the game file steam is downloading.

Ill post a screenshot when I have downloaded the last few games which should be roughtly 20 GB data, I had to reboot because the computer was literally unuseable with 99 % memory usage over 32GB, couldn't even take a screenshot of that.



Also, I checked now by quitting steam and then restarting it.
Not even after that, Steam releases memory back. Steam actually hard allocates memory for no apparent reasons, and wont even release it back after QUITTING steam.

[Sebastian Nielsen]

Here is a pic. Just look how much memory Steam has requested and not even given back to the OS after Steam is quitted completely.

http://steamcommunity.com/sharedfiles/filedetails/?id=572810814

[henryg]

> Quitting steam also wont release the claimed memory.

We cannot reproduce this here, we do not see this behavior happening. And the fact that the memory is not released when Steam exits makes me think that it is some other driver or operating system component that is local to your computer which is causing the issue. Do you have any way to verify this on another system?

[henryg]

Also, I do not think that you are interpreting those numbers correctly. The operating system uses all available free memory as a disk cache. If you download a 50 GB game and you have 64 GB of memory, the OS may use up to 50 GB to cache the downloaded game files. But as soon as another application request memory, the OS will trim the file cache and return that physical memory to the application just as if it had been free.

[Sebastian Nielsen]

The green memory space is never released to the system. I can try to re-verify the issue by deleting some large games and redownloading them with a android set on timelapse to photograph the screen physically (to ensure only Steam is running at that time and no screen recording program) and then I can submit a movie.

I have no possibility to verify this on a another computer, but since this only happens when I have a active download queue with a large amount of large games, this issue can be isolated to Steam.

[Balderick]

I tried looking at memory usage using resource monitor and did not understand the blue portion of memory allocation. It is really avaialble memory basicaly - i think or that is the way i read it.

I tried to reproduce what you describe but can not either on a machine with only 8GB of ram.

Have you tried using task manager instead to monitor memory usage?

[Sebastian Nielsen]

The thing here is that I got the warning dialog, here is a picture on how it look (not from my computer, was unable to take screenshot due to the high memory usage):
http://emby.media/community/uploads/inline/5228/55a8563118eb7_Capture.png

This on a 32 GB computer!

And in that dialog, it said "Steam client Bootstrapper"

[Balderick]

You could upload that to your profile as artwork which would then allow you to inline it here by just posting the url of the artwork.

[Sebastian Nielsen]

Here is a timelapse, shot in 1800x, of Steam downloading. (Real time: 3.5 hours) (Sorry for the bad angle, had to place the phone in a improvised stand in a really avkward position)
https://www.youtube.com/watch?v=bIx3YZXh0rE

This is a Completely clean run, with the computer just booting, and then I just told steam to download some heavy games. Steam just crash itself due to insufficent RAM. It leaks RAM as hell. I even exited Skype to ensure theres no software distrurbing the mesurement process. This ONLY happens when I download on Steam, and it won't even release the memory if I exit steam.

The green bar will stay allocated and unavailable for other programs, regardless of if Steam exits by crash or exits by user interaction.
Only way to resolve issue is to reboot computer.

That I can reproduce the issue by just downloading games through Steam, is clear evidence that this issue is originating from Steam and not some other application from my computer.

Here is the result in the form of a picture.
http://steamcommunity.com/sharedfiles/filedetails/?id=573085743


Edit: Tried it again. Same result everytime. Everytime I fire up a download of some heavy games, it causes the computer to cry.
Now the third time I tried, I got this:
http://steamcommunity.com/sharedfiles/filedetails/?id=573154399

Clearly its Steam causing it.

[henryg]

That video is crazy! We have never seen anything like that before. And in fact, it should not be possible for the Steam process to allocate that much memory, because it is a 32-bit program and is limited to 3 GB of total memory by the operating system. We have been downloading games all day long on multiple computers here at the office, and our systems still have almost all of their memory free.

I really feel like this must be a bad interaction or bug with some other piece of software, like a virus scanner or a firewall program, or a network card driver (because of the downloading?).

I did a search for "windows 10 high memory usage until reboot" and found several internet pages which mention a "Killer network driver" as a problem:

http://superuser.com/questions/949244/windows-10-high-memory-usage-unknown-reason

http://www.tenforums.com/performance-maintenance/14595-very-high-memory-usage-system.html

https://www.reddit.com/r/Windows10/comments/369uwk/does_windows_10_have_a_system_wide_memory_leak/

If you have a motherboard with a "Killer" network port, try updating the drivers from the official Killer website?

[Tucu]

That is very similar to a memory leak when streaming to the Steam Link that we saw a few days ago.
It happened with the latest Killer drivers, but it can be fixed with a registry change to disable the Windows Network Usage Data Monitor:

The solution for this says to set 'Start' to 4 in your registry under
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Ndu\


This is the thread where the problem and the solution was discussed:
http://steamcommunity.com/app/353380/discussions/0/490125737474451406/

[Sebastian Nielsen]

henryg:
I should check that Killer Network card driver, because I actually have a killer network card on that Gigabyte GA-X99M GAMING 5.
But I find it strange that a WHQL-certified network driver, that is pushed officially through Windows Update, and even included on the Windows 10 TH2 install media, can leak memory. As far as I know, it wouldn't pass WHQL certification if it leak memory.

I even checked who did signed the driver, both the Killer Network filter driver and the main driver: "Microsoft Windows Hardware Compatibility Publisher", which does chain with the Microsoft Root CA.

But I should update the driver, and also apply Tucu's fix.
After that, I should try downloading some heavy games again and see if this issue persist. If the issue persist, there might be some race condition with Steam that we need to investigate further.

About antivirus/Firewall:
The only antivirus Im running is the Windows own antivirus, called Windows Defender. I know that Windows Defender is pretty efficient with the resources and thats why im using it, so theres no memory leak there.
And the firewall, I of course use the built-in Windows firewall. (In combination with a firewall appliance that is called PfSense, but that is a hardware appliance that is connected between the router and the internet RJ45 socket in-wall, and could, impossibly affect memory usage on a client)

[Sebastian Nielsen]

henryg:

The issue turned out to be the Killer network card. Download the base drivers (without killer features) from their website, and then spend some countless hours of uninstalling the Killer Network Manager (that was basically everywhere since Windows Update did behave like killer network manager hit the fan and spread everywhere inside the computer HDD and in registry).

Also, set:
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Ndu\Start to "4".

After that, reboot computer, issue solved.

Have now verifyed and I see no larger usage than 2.5 GB, even when downloading about 150GB games.


Thank you for finding out this for me. For me it just looked like 100% a steam issue.
However, I find it very strange that WHQL-certified drivers can behave in this way, my tought was that it was impossible for a leaking driver to be WHQL-certified, which is mandatory to allow install in x64 mode on a x64 system.


A great idea could be to write a steam support article about this and push it to the top, like "If you notice high memory usage or memory leaks during using Steam Link or downloading/installing games, this can be a issue of the default WHQL-drivers and the Network Manager application that Microsoft supplies for computers having a Killer Network Card".

And then full instructions on how to remove network manager and installing the correct drivers.

[[oRLY]Alia5]

Whql isn't a guarentee for anything... Also drivers are still Software
And bugfree Software is hard to make... Very hard indeed