Steam Web API

[D3XX3R]

How are websites like steamdb(dot)info or steamid(dot)uk collecting so many data from every single user on Steam? Do they have a special API key without limits or do they have a database full of API keys?

[cSg|mc-Hotsauce]

They aren't collecting data for "every" user on Steam. They only scrub public info.

[D3XX3R]

But in the case of steamid(dot)uk they are collecting user info.
I assume they collect data as soon as a user on their site requests them.
But they are still limited with 100000 requests per day.

[cSg|mc-Hotsauce]

Again, only public info.

Only my Artwork, screenshots, forum posts and reviews are public. That is all they can scrub, if they even care to.

[MalikQayum]

Origineel geplaatst door cSg|mc-Hotsauce:

Again, only public info.

Only my Artwork, screenshots, forum posts and reviews are public. That is all they can scrub, if they even care to.

:qr:

I don't get what you are trying to tell the OP, also i think you mean scraping but even that makes no sense, none of the sites he mentioned does scraping they use the api.
Even if they don't use the web api, making 100k + requests like that will get you a temp ip ban so clearly that is not what they would be doing.

Origineel geplaatst door Koray | d3xx3r.de:

How are websites like steamdb(dot)info or steamid(dot)uk collecting so many data from every single user on Steam? Do they have a special API key without limits or do they have a database full of API keys?

the 100.000 request limit per day can change if valve approves it, what kind of arrangement some sites have is between valve and the site.

your assumption is that steamid(dot)uk and sites of that sort actually crawl the profiles is wrong, well maybe some do but i am fairly confident that most these sites don't crawl.

so the data you might be seeing on these site, is "old" data they have stored in their db, so every time you visit a profile on their page they might make another request to update that data or you would actually have to make a search for the said profile on their site to update that specific steamids data in their db.

steamdb did crawl profiles for a while but they were most like one to be approved by valve to up their request limits.

I do get the impression you underestimate how many request 100k really is but it is a lot of requests.

[D3XX3R]

Thanks for your answer MalikQayum.

I guess the only logical explanation is that Steam and those sites have an arrangement as you stated it out.

But I'm currently having an account database with 167 accounts in it. I'm crawling data about those accounts every 5 minutes.
I'm crawling data from ISteamUser/GetPlayerSummaries and ISteamUser/GetPlayerBans.
So 167(accounts)x288(every five minutes)x2(two APIs to call) equals 96.192 requests per day.
As soon as I'm reaching 174 accounts in my database I'm above the limit.

That's why I wanted to know how other greater websites do it.
I should probably not query every 5 minutes to lower my requests to the API.

[cSg|mc-Hotsauce]

Origineel geplaatst door Koray | d3xx3r.de:

I guess the only logical explanation is that Steam and those sites have an arrangement as you stated it out.

Not 100% correct.

You are limited to one hundred thousand (100,000) calls to the Steam Web API per day. Valve may approve higher daily call limits if you adhere to these API Terms of Use.

Read https://steamcommunity.com/dev/apiterms

[MalikQayum]

Origineel geplaatst door Koray | d3xx3r.de:

Thanks for your answer MalikQayum.

I guess the only logical explanation is that Steam and those sites have an arrangement as you stated it out.

But I'm currently having an account database with 167 accounts in it. I'm crawling data about those accounts every 5 minutes.
I'm crawling data from ISteamUser/GetPlayerSummaries and ISteamUser/GetPlayerBans.
So 167(accounts)x288(every five minutes)x2(two APIs to call) equals 96.192 requests per day.
As soon as I'm reaching 174 accounts in my database I'm above the limit.

That's why I wanted to know how other greater websites do it.
I should probably not query every 5 minutes to lower my requests to the API.

Your setup does seem a bit off, i would not make request at all that frequently.
You also clearly doing the requests wrong, as both GetPlayerBans and GetPlayerSummaries allows you to make 1 request with up to 100 ids by the comma delimiter, i would recommend you do that instead.

https://api.steampowered.com/ISteamUser/GetPlayerBans/v1/?key=<apikey>&steamids=<steamid1>,<steamid2>

https://api.steampowered.com/ISteamUser/GetPlayerSummaries/v1/?key=<apikey>&steamids=<steamid1>,<steamid2>

it would also help a bit more if you described a bit about what you want to do, if you want some proper feedback and advise on how to maybe improve your website.

[D3XX3R]

Origineel geplaatst door MalikQayum:

Your setup does seem a bit off, i would not make request at all that frequently.
You also clearly doing the requests wrong, as both GetPlayerBans and GetPlayerSummaries allows you to make 1 request with up to 100 ids by the comma delimiter, i would recommend you do that instead.

https://api.steampowered.com/ISteamUser/GetPlayerBans/v1/?key=<apikey>&steamids=<steamid1>,<steamid2>

https://api.steampowered.com/ISteamUser/GetPlayerSummaries/v1/?key=<apikey>&steamids=<steamid1>,<steamid2>

You are totally right MalikQuayum. I totally forgot that you are able to get data from up to 100 accounts in one request.

[MalikQayum]

Origineel geplaatst door Koray | d3xx3r.de:

Origineel geplaatst door MalikQayum:

Your setup does seem a bit off, i would not make request at all that frequently.
You also clearly doing the requests wrong, as both GetPlayerBans and GetPlayerSummaries allows you to make 1 request with up to 100 ids by the comma delimiter, i would recommend you do that instead.

https://api.steampowered.com/ISteamUser/GetPlayerBans/v1/?key=<apikey>&steamids=<steamid1>,<steamid2>

https://api.steampowered.com/ISteamUser/GetPlayerSummaries/v1/?key=<apikey>&steamids=<steamid1>,<steamid2>

You are totally right MalikQuayum. I totally forgot that you are able to get data from up to 100 accounts in one request.

I will still say requesting the data through the api like this is definitely not advised imo, as you will sooner or later run into the same issue you are facing now.
All you really do is prolonging the inevitable which is you are going to reach the api request limits for that api key.
Solution to that would be to generate more api keys to different steam accounts, and continue this cycle.

or you should look into making multiple of cronjobs (assuming you doing this on your webhost) and have the php scripts check for when you last updated something and get the oldest entries that have not yet been updated, then limit the query to 100 ids.

[D3XX3R]

I would prefer to use the 2nd method. But I have yet no idea how to do that.

[D3XX3R]

Could you help me out a little bit? I have no idea to realize this in my cronjob.
Normally I would do a while loop for every single database entry. But I really have no idea how to do this. I've added you on Steam.

[MalikQayum]

first you would have to make sure your insert query has a column "last_updated" and insert the date as below.

INSERT INTO table ( field1, field2, last_updated)
   VALUES
   ( value1, value2, NOW());

then you want make a query where you get 100 rows, where the last_updated is one day less than the current date.

$date = date('Y-m-j');
$newdate = strtotime ( '-1 day' , strtotime ( $date ) ) ; // I would do something -1 day 
$newdate = date ( 'Y-m-j' , $newdate );

$query = $db->query("SELECT * FROM table WHERE DATE(last_updated) <= DATE('" . $newdate . "') ORDER BY last_updated ASC LIMIT 100");

writing this you probably should not do an insert but insert on duplicate key update query but you can figure that out yourself i guess.

[D3XX3R]

Already got that working. But how do I lineup all the SteamIDs in the request and handle them?

[MalikQayum]

Origineel geplaatst door Koray | d3xx3r.de:

Already got that working. But how do I lineup all the SteamIDs in the request and handle them?

you mean iterating through the json ?
just use a foreach loop

[D3XX3R]

Got it working perfectly. Thank you very much for your help.