Get item_nameid for steam items

[zig]

Is there anyway i can get or convert some other parameters to item_named? I want to use it for getting item histogram data using this
https://steamcommunity.com/market/itemordershistogram?country=PK&language=english&currency=1&item_nameid=176096390&two_factor=0&norender=1

But there is no mention of item_nameid anywhere on internet about how one can get it or convert to it.

[MalikQayum]

the item_nameid is only available on the listing page itself. You will have to scrap the data and store it in your own local or remote database to easily look it up again, to avoid making an unnecessarily request in the future.

[zig]

Originally posted by MalikQayum:

the item_nameid is only available on the listing page itself. You will have to scrap the data and store it in your own local or remote database to easily look it up again, to avoid making an unnecessarily request in the future.

So basically i will have to scrap all the 15000 pages ? do you know any database that already exist by any chance ?

[MalikQayum]

the above item_nameid which appid does it belong to and is that the appid you want to be focusing on?
I do not know of any public database, sry but making one shouldn't take long.

[zig]

Originally posted by MalikQayum:

the above item_nameid which appid does it belong to and is that the appid you want to be focusing on?
I do not know of any public database, sry but making one shouldn't take long.

this above item belong to appid 730 which is cs go. Yes thats the only appid i need item_nameid for. Only problem is steam only allow few requests and after that you are blocked for several minutes. It took me almost 8 minutes to make 40-50 requests for getting hash names and some other data from market listings. Also you know anyway once i have the item_nameid that i can make request in batch instead of getting data for single item at a time lets say get for 100 items at a time?

[MalikQayum]

https://steamcommunity.com/market/search/render/?query=&start=0&count=10&search_descriptions=0&sort_column=popular&sort_dir=desc&appid=730&norender=1

we can use these 3 things to create something that will use less requests.

total_count which we get from the first request
and the 2 parameters in the url:
&start=0
&count=10

{
    "success": true,
    "start": 0,
    "pagesize": 10,
    "total_count": 15042,
    "searchdata": {
        "query": "",
        "search_descriptions": false,
        "total_count": 15042,
        "pagesize": 10,
        "prefix": "searchResults",
        "class_prefix": "market"
    },
    "results": [......]

the max count seems to be 100 so we can can change it to that.
&start=0
&count=100
which makes this so much easier as we would only need to do 100 / 15000 = 150 requests and only do those request once.
so create a cronjob to do it once every 1 min and you can have it in 15 min or so.

on the topic of item_nameid, there is no way around having to go through each items page.
my suggestion would be to create a separate cronjob just for that and have the script do a request every 30 sec, maybe even 1 min.

which would 60 x 24 = 1440 a day which makes this 15000 / 1440 = around = 11 days.

when done, you have a base and all you then really need to do is keep adding new items to the db tbl, as cs items gets added to the game.

you might want to create a detection or just use the first thing we did and add new items we haven't seen before to the tbl and the second cronjob should then eventually get to that item.

[Chrys2015]

Originally posted by ZIGGY-Zz:

Is there anyway i can get or convert some other parameters to item_named? I want to use it for getting item histogram data using this
https://steamcommunity.com/market/itemordershistogram?country=PK&language=english&currency=1&item_nameid=176096390&two_factor=0&norender=1

But there is no mention of item_nameid anywhere on internet about how one can get it or convert to it.

[zig]

Originally posted by MalikQayum:

https://steamcommunity.com/market/search/render/?query=&start=0&count=10&search_descriptions=0&sort_column=popular&sort_dir=desc&appid=730&norender=1

I am already using this to get list of all items withing specific price range. The problem with this is that it only scraps information from market listing pages. You dont get item_nameid through this nor the buy orders and histogram. Only useful thing through this is that you can get items withing specific range, their minimum price and market_hash for the item.

Next use the market hash for each item to get xhr request from that item page which look something like this
https://steamcommunity.com/market/itemordershistogram?country=PK&language=english&currency=1&item_nameid=176096390&two_factor=0&norender=1
and extract name id from this to build the database. Which mean 11 days for building database. But even after that i will have to scrap for days to get the details for required items. I was hoping to make like 2-3k request in a day for what i am doing since histogram can change quite a lot even in few days

But thank you any way for your help.

[MalikQayum]

maybe the way you want to do it is not the right way or you might not really understand the way i am trying to tell you how to set it up with.

so which language do you use for scripting/programing ?

[zig]

Originally posted by MalikQayum:

maybe the way you want to do it is not the right way or you might not really understand the way i am trying to tell you how to set it up with.

so which language do you use for scripting/programing ?

Currently i am using python for scripting. Although in the past i have used various other languages such as C/C++, Java,Verilog,MATLAB mainly for various other purposes. But now that i am trying learn web development and some related skills i thought instead of learning javascript i can use python and get started fast instead of having to wait another month or something for learning it first

As for the query you explained can i get item_nameid from that by some modifications. Because currently it does not have that data.

[MalikQayum]

the concept of what i wanted you to do is separate each task.
first you would create an entire db of the item itself. which would be done by the cronjob that handles that.
then i would have a second cronjob just for add item_nameid that it scraps from each page and adds it in the right column.

now i have the data i want to build my page with.
i can fetch the item_nameid from my own tbl and add it to the page i want the histogram to display the data on.

seems pretty straightforward to me, i do not work in python but this probably won't even take much time to set up.

[zig]

Originally posted by MalikQayum:

the concept of what i wanted you to do is separate each task.
first you would create an entire db of the item itself. which would be done by the cronjob that handles that.
then i would have a second cronjob just for add item_nameid that it scraps from each page and adds it in the right column.

now i have the data i want to build my page with.
i can fetch the item_nameid from my own tbl and add it to the page i want the histogram to display the data on.

seems pretty straightforward to me, i do not work in python but this probably won't even take much time to set up.

Yes i am almost doing it the same way you had mentioned here and the tasks are also separated. Only difference is i put it sleep if i get a network error or return status other than 200 and try again after a minute. The script almost all setup only problem is number of requests because i want fresh data for 2-3k items every day atleast. And without spending on proxies i dont think 2-3k requests are possible

[MalikQayum]

explain to me what data in the itemordershistogram you need.

[zig]

Originally posted by MalikQayum:

explain to me what data in the itemordershistogram you need.

Buy orders list, sell order list and sell history

[MalikQayum]

I don't want to assume the worst but this just points in one direction. This is the kind of data one would want if they were to build a market bot that buys and sell for small margin profit.

The question is what are you making, if not that? because if it is that then we should writing about it here is not so wise because this goes against the SSA/TOS as you are not allowed to automate market transactions.

there is a reason why there is no api or way to query multiple of items at once, as it is clearly a way for valve to prevent automation.
(which in hindsight now that i think of it explains why they even have the item_nameid.)

but if this is not the case then feel free to explain in details why you need that data then maybe there might be a solution.

[zig]

Originally posted by MalikQayum:

I don't want to assume the worst but this just points in one direction. This is the kind of data one would want if they were to build a market bot that buys and sell for small margin profit.

The question is what are you making, if not that? because if it is that then we should writing about it here is not so wise because this goes against the SSA/TOS as you are not allowed to automate market transactions.

there is a reason why there is no api or way to query multiple of items at once, as it is clearly a way for valve to prevent automation.
(which in hindsight now that i think of it explains why they even have the item_nameid.)

but if this is not the case then feel free to explain in details why you need that data then maybe there might be a solution.

Basically this a learning project about web scrapping. I do not wanna make a bot to automate buying and selling. I know that those bots get banned easily and i wouldn't waste my money on them knowing that. But i do have multiple reasons ,using item popularity, sell history and number of items i wanna try making a program to generate analytics how the price tend is expected to move (in short term). Since i been using steam for quite a while and is quite familiar with it thought better try with this.