Steam data scraping

[zig]

Hi i am working on a little project for the sake of learning data scraping. I am trying it on steam. First thing is it allowed on steam? How many request we can make per unit of time? Also whenever i try to make request on market list with descending price for example https://steamcommunity.com/market/search?appid=730#p150_price_desc instead what i get is
https://steamcommunity.com/market/search?appid=730#p150_popular_desc the _price_desc parameter is not working. Any possible solution to this?

[wuddih]

data scraping?
what you wanna do looks more like screen scraping.

if you wanna do data scraping, look what your browser does when it calls that url with that anchor parameter.

i dont wanna explain the whole shnigshnag how to get to this data that you wanna use
https://steamcommunity.com/market/search/render/?query=&start=1490&count=10&search_descriptions=0&sort_column=price&sort_dir=desc&appid=730&norender=1

you will figure that out yourself, how i got that.

as long as you dont do it from 150 servers with 10k requests per second, they dont care a flip flop.
but due to respect of the service and webrequest limiters that will ban your ip, you might wanna throttle your query limit to about 10-15 per minute.

[zig]

Автор сообщения: wuddih

data scraping?
what you wanna do looks more like screen scraping.

if you wanna do data scraping, look what your browser does when it calls that url with that anchor parameter.

i dont wanna explain the whole shnigshnag how to get to this data that you wanna use
https://steamcommunity.com/market/search/render/?query=&start=1490&count=10&search_descriptions=0&sort_column=price&sort_dir=desc&appid=730&norender=1

you will figure that out yourself, how i got that.

as long as you dont do it from 150 servers with 10k requests per second, they dont care a flip flop.
but due to respect of the service and webrequest limiters that will ban your ip, you might wanna throttle your query limit to about 10-15 per minute.

I was trying to get item hash for items over specified price ranges . I was getting link associated with elements having class = "market_listing_row_link" and spliting them up to get hash such as USP-S%20%7C%20Cortex%20%28Field-Tested%29 from https://steamcommunity.com/market/listings/730/USP-S%20%7C%20Cortex%20%28Field-Tested%29

And then doing additional query to using that hash to get price and some other info. (but i wouldnt get it in specified range because sort wasnt working)

But using your method is much better and provide far more info for way less work. Although i do not understand now how you got this query but i think i will after some search.
So thank you very much for the help

p.s : explained the method i was using so that if you have another tip i would be very glad

[wuddih]

f12 opens browser console, it is xhr request.

as for why your call did not work, probably:

the url is called, then the anchor is ajax handled by the javascript on the page and content is changed upon that result.

1st: https://steamcommunity.com/market/search?appid=730 with default parameters
2nd: #p150_price_desc <- call the ajax for this stuff and update the "table" with new results.
(the same request i did above, just without the norender parameter)

you were stuck with the first, i assume because what you coded did not "wait" for the 2nd thing to happen as this requires a "real webbrowser" and depending in what you code in this can be a pain in the butt to get to work.

you were assumingly also not getting page 150 of popular sorting, but default page 1, otherwise i would not know how you managed to do that.

[zig]

Автор сообщения: wuddih

f12 opens browser console, it is xhr request.

as for why your call did not work, probably:

the url is called, then the anchor is ajax handled by the javascript on the page and content is changed upon that result.

1st: https://steamcommunity.com/market/search?appid=730 with default parameters
2nd: #p150_price_desc <- call the ajax for this stuff and update the "table" with new results.
(the same request i did above, just without the norender parameter)

you were stuck with the first, i assume because what you coded did not "wait" for the 2nd thing to happen as this requires a "real webbrowser" and depending in what you code in this can be a pain in the butt to get to work.

you were assumingly also not getting page 150 of popular sorting, but default page 1, otherwise i would not know how you managed to do that.

I understand it very well now thank you very much. Also you were right since i were only trynna get first few pages during test run i did not notice but now that i look at data it's exactly how you said it.

[Sazzouu]

Sure it is absolutely legal to do so. Valve knows their business in terms of traffic control (per user - don't troll me with Summer Sale now) so if you cause severe traffic as only one user Valve knows how to restrict you there.

Anyways they actually deliver several features with their own API for data scraping...
https://steamcommunity.com/dev
For further information dig through here...
https://developer.valvesoftware.com/wiki/Steam_Web_API

Those APIs as far as I can tell do not deliver market features but it shows that Valve not only allows data scraping but also encourages it. There are lots and lots of useful marketing-tools that are totally accepted by Valve. Best example is http://steamdb.info

As for your example request in your OP the Price-descending feature works on my end. Probably a mistake on your request then? I've written some tools to dig into the markets myself to keep track of my items of interest such as gems and booster packs.

[zig]

Автор сообщения: BeatZ #GrandPrix2.0Please

Sure it is absolutely legal to do so. Valve knows their business in terms of traffic control (per user - don't troll me with Summer Sale now) so if you cause severe traffic as only one user Valve knows how to restrict you there.

Anyways they actually deliver several features with their own API for data scraping...
https://steamcommunity.com/dev
For further information dig through here...
https://developer.valvesoftware.com/wiki/Steam_Web_API

Those APIs as far as I can tell do not deliver market features but it shows that Valve not only allows data scraping but also encourages it. There are lots and lots of useful marketing-tools that are totally accepted by Valve. Best example is http://steamdb.info

As for your example request in your OP the Price-descending feature works on my end. Probably a mistake on your request then? I've written some tools to dig into the markets myself to keep track of my items of interest such as gems and booster packs.

Steam should provide some market APIs. Anyway thanks for all the info. Also by any chance your tools available on github?