Steam Hardware Survey Fix - Overcounting of Asia Cybercafes

[Satoru]

https://store.steampowered.com/hwsurvey

So many users would have noticed about August last year, the popularity of PUBG in China seemed to push the Steam Hardware Survey results to a massive increase in Windows 7 and Simplified Chinese language users.

Steam has identified that this was in fact a mis-count in the survey causing cyber-cafe computers to be counted multiple times during the survey causing the massive skew in their data.

As of this month they've implemented some changes that prevents cyber-cafes from being counted multiple times. You already see that the largest skews, Windows 7 and Simplified Chinese, have come down this month. Expect that these will likely drop further in the coming months and be more in line with the kind of data that was available in August of 2017.

STEAM HARDWARE SURVEY FIX – 5/2/2018
The latest Steam Hardware Survey incorporates a number of fixes that address over counting of cyber cafe customers that occurred during the prior seven months.

Historically, the survey used a client-side method to ensure that systems were counted only once per year, in order to provide an accurate picture of the entire Steam user population. It turns out, however, that many cyber cafes manage their hardware in a way that was causing their customers to be over counted.

Around August 2017, we started seeing larger-than-usual movement in certain stats, notably an increase in Windows 7 usage, an increase in quad-core CPU usage, as well as changes in CPU and GPU market share. This period also saw a large increase in the use of Simplified Chinese. All of these coincided with an increase in Steam usage in cyber cafes in Asia, whose customers were being over counted in the survey.

It took us some time to root-cause the problem and deploy a fix, but we are confident that, as of April 2018, the Steam Hardware Survey is no longer over counting users.

[Start_Running]

I'm not sure that's actually over counting. No more than say you or eye get over counted when we run the steam client from different PCs.
I'm all for this shift since it might get developers aiming juuust a bit lower.

[Satoru]

Originally posted by Start_Running:

I'm not sure that's actually over counting. No more than say you or eye get over counted when we run the steam client from different PCs.
I'm all for this shift since it might get developers aiming juuust a bit lower.

Its an over count because basically what I suspect happened was that cyber cafes were wiping the system weekly/daily or something. This means that

1) user A walks up to computer A, gets the survey
2) Computer A gets wiped
3) user B walks up to computer A, gets the survey because it looks 'new'
4) Computer A gets wiped
5) user C walks up to computer A, gets the survey because it looks 'new'
6) Computer A gets wiped

Repeat ad-infinitum

Basically the exact same computer was getting counted multiple times. Now multiply that by every computer, in every cybercafe in China. And you can start seeing where this skew comes from.

[Snapjak]

Wow that sure altered the stats compared to just a few days ago.

Good altered, mind you.

[Satoru]

Originally posted by Snapjak:

Wow that sure altered the stats compared to just a few days ago.

Good altered, mind you.

China still remains a fairly large market, just not "50% of steam users are chinese" important (its still a top 3 language English/Chinese/Russian)

Interestingly Win10 adoption is above 50% now which it wasn't before. We'll have to see in the next few months where that evens out to

[Snapjak]

Hopefully it remains relatively similar with the usual variances for new hardware, new markets, and whatever flavour of the week game is on sale dictating where the masses go.

[Satoru]

Originally posted by Snapjak:

Hopefully it remains relatively similar with the usual variances for new hardware, new markets, and whatever flavour of the week game is on sale dictating where the masses go.

The steam survey has been pretty good with that, and i beleive that steamspy (before it was blocked) at least seemed to show that overall the geographical/language part of the steam survey matched up with their results.

I exepct, barring another wierd anomaly, that the steam survey will continue to be a good resource for trends for gaming systems

[cSg|mc-Hotsauce]

Cafes should be counted as 1 and not get the random survey for more than 1 account that is using those PCs. I agree.

[Snapjak]

Now if we could just get Valve to update some of the parts to more modern amounts (looking at you network speeds).

[cSg|mc-Hotsauce]

And 99.87% with mics?

[Count_Dandyman]

Originally posted by cSg|mc-Hotsauce:

Cafes should be counted as 1 and not get the random survey for more than 1 account that is using those PCs. I agree.

:qr:

The survey is meant to get an idea of what is being used by accounts so honestly it would be a heavily flawed and pointless survey if they deliberatly cut those accounts out of the survey.

[Satoru]

Originally posted by Count_Dandyman:

The survey is meant to get an idea of what is being used by accounts so honestly it would be a heavily flawed and pointless survey if they deliberatly cut those accounts out of the survey.

The fix doesn't appear to cut out cyber cafes entirely. It prevents a single computer from being part of the survey multiple times a year

[Count_Dandyman]

Originally posted by Satoru:

Originally posted by Count_Dandyman:

The survey is meant to get an idea of what is being used by accounts so honestly it would be a heavily flawed and pointless survey if they deliberatly cut those accounts out of the survey.

The fix doesn't appear to cut out cyber cafes entirely. It prevents a single computer from being part of the survey multiple times a year

yeah but that idea is still flawed as when you are looking for stats on what users are gaming on blocking a system that could be the only one used by multiple accounts from showing up more then once will just skew the results too heavily toward systems that are only used by one person and then you would see choices being made based on people thinking the steam userbase has 50%+ people with a certain setup when the reality could be closer to 5%.

[Satoru]

Originally posted by Count_Dandyman:

Originally posted by Satoru:

The fix doesn't appear to cut out cyber cafes entirely. It prevents a single computer from being part of the survey multiple times a year

yeah but that idea is still flawed as when you are looking for stats on what users are gaming on blocking a system that could be the only one used by multiple accounts from showing up more then once will just skew the results too heavily toward systems that are only used by one person and then you would see choices being made based on people thinking the steam userbase has 50%+ people with a certain setup when the reality could be closer to 5%.

Functionally its a hardware survey

Not a user survey

That's a big difference which is why your point would only be valid if it was designed as a user survey.

Its meant to quantify the kind of hardware that steam runs on, not what users are out there.

[Start_Running]

Originally posted by Satoru:

Originally posted by Start_Running:

I'm not sure that's actually over counting. No more than say you or eye get over counted when we run the steam client from different PCs.
I'm all for this shift since it might get developers aiming juuust a bit lower.

Its an over count because basically what I suspect happened was that cyber cafes were wiping the system weekly/daily or something. This means that

1) user A walks up to computer A, gets the survey
2) Computer A gets wiped
3) user B walks up to computer A, gets the survey because it looks 'new'
4) Computer A gets wiped
5) user C walks up to computer A, gets the survey because it looks 'new'
6) Computer A gets wiped

Repeat ad-infinitum

Basically the exact same computer was getting counted multiple times. Now multiply that by every computer, in every cybercafe in China. And you can start seeing where this skew comes from.

I never assumed it to be that sopjistricated as in remembering the machines differently from the users. I each account is assumed to be a unique sample.

And perhaps this is more accurate. The purpose of the survey isn't so much to identify systems as it is to determine what sorts of systems users have access to and use. In which case the way cyber cafe's are counted seems legit. if the only machinbe some of these users have access to is the cyber cafes then it's strill accurately capturing the date they're after.,

What are our users gaming on?

[Radene]

Originally posted by Count_Dandyman:

Originally posted by Satoru:

The fix doesn't appear to cut out cyber cafes entirely. It prevents a single computer from being part of the survey multiple times a year

yeah but that idea is still flawed as when you are looking for stats on what users are gaming on blocking a system that could be the only one used by multiple accounts from showing up more then once will just skew the results too heavily toward systems that are only used by one person and then you would see choices being made based on people thinking the steam userbase has 50%+ people with a certain setup when the reality could be closer to 5%.

Good point.

So does valve want to survey "systems" or "users"? That's the question here.

Does it want to know whether 50% of all systems run win10, or whether 50% of all users play on win10?

The fact that this thread exists is solid proof that there's a statistically significant difference between the two.