Building a successor to Barter

[Nao 尚]

If you aren't aware of it, Barter.vg's administration hasn't been on Steam or Barter in many months now. The website's domain may or may not expire come November, hosting may even expire before that, making us all users without a 'good' trading site to use. Unfortunately, the source code for the website remains a secret, so it's not like anyone can just magically take it to another server and ensure its well-being.

So I've been talking with many people, and it looked like I had the most motivation (and/or skill) to try and build a successor website.

The idea is to have something that's close in spirit to Barter, allowing you to retrieve your custom wishlist & tradables, and make offers to other users. I'll also try to add more 'community' features, because why not. There is little money to be made on this website, only the assurance that we can keep the idea alive. So if I ever lose motivation, I'll have to count on someone else to take over.

This will be a community effort, so I figured I should open a new topic here on the Barter Steam community, just so we can discuss current, future and removed features.

[Nao 尚]

Publicado originalmente por r6d2:

These problems occur on the Valve database all the time. They are mostly due to developers not being experienced enough on the spirit on the database and the canonical way things should be implemented. And paraphrasing what Larry Wall said about Perl, "With Valve there's more than one way to do it" :-)

Yeah, sometimes in game forums I see people telling authors how to configure their Steam store pages... lol. (e.g. Free to play vs. Free, for instance.)

I noticed from your post on the Bundle thread that you're not familiar with the database, so let me try to roughly describe the structure as I know it:

Yeah, I'm familiar with the basics, but it goes really deep...

Every game you buy on Steam/Retail/Bundle is actually a SubID (sub originally comes from "subscription"). You NEVER get to buy/activate an AppID on its own, but always a SubID instead.

How do you know that for sure..? to me, it seemed that if a parent was set to '0', it didn't belong in a sub...

Just for the sake of completeness of this rough schematics, there's another type of entry called Bundle, which is also implemented via a SubID, but just like the package, is conceptually different from the subscription. I've observed that Bundles are even more volatile than packages, since they are tailored to your own account.

For the life of me, I don't see myself adding those to the site... I mean, values are fucked up by this. (Although it's unlikely these prices will be lower than the lowest Steam price ever. Also, what about coupons? Ah ah... How do we know there are 90% coupons for a game? If it's hard to get them, does it mean they shouldn't influence the value..? Etc.)

Barter syncs from user's accounts the AppIDs on Library and Wishlist, not the SubIDs. The link between AppIDs and SubIDs is not available on Barter and that's why the admin used to create the packages one by one on request.

Oh... On request. That explains the inconsistencies, I guess!

Now, regarding the database, here's where I'm getting to a stop... If an app doesn't have a Steam store page, then the Store API returns 'false' for it. A simple example would be the DLCs for Spec Ops The Line, a recent Humble bundle tier 1. It not only has regular DLCs without a store page (e.g. 'Coop Challenge Mode', app ID of 203180), it also has dummy DLCs named like 'ValveTestApp203182', which I'm pretty sure are just there to use space in the database... Pff.

Thing is, Depressurizer has Coop Challenge Mode, and it clearly says its parent has ID 50300 (Specs Ops The Line), so it's information that's *better* than what I could get from the Store API, or even from scraping the store itself AFAIK, because 'Coop Challenge Mode' was included in the app, so I think it's one of those automatically included DLCs that don't show up on the store at all...
I'm not sure where they got this from, really. But it makes it harder for me to just say, "okay, I'll stick to scraping"... What do you think?

[Nao 尚]

Hmm, yep, just logged out of Steam, checked Spec Ops, and that DLC isn't showing up, so there's no way to even know it exists. Maybe it used to have its own page and that's when Depressurizer scraped it, I don't know...
Then again, it's probably for the best: if the DLC is included in the game, there's no reason to have it in one's tradable list, of course.
Staying with the scraping idea for now.
(Also need to consider sending $_POST data to confirm age for games that require an age check...)

Publicado originalmente por r6d2:

For example, this is a typical situation that shows why change is something you have to live with on a site like Barter. An episodic game, let's say "Kakuka", evolves like this:

"Kakuka - Episode 1" is released and gets its own AppID, say 111

It's usually a multiple of 10. Just to 'leave space' for 9 DLCs or demos or trailers after that.

Individual episodes will continue to be available for a while, but at some point they will just redirect to app/120 as well. Some developers prefer to make ALL individual episodes to be renamed to "Kakuka Season 1 Deluxe", making the matter even more confusing.

Yeah... There are some really fucked up games when it comes to episodic.
For instance, Life is Strange... Now that episode 1 is free, does this mean it doesn't drop cards anymore, and only the 2-5 pack does? Are the cards still attached to the same app ID? ;) Older Telltale games... They clutter my library with episode numbers, when they should really be using DLCs for individual episodes. But, of course, even the DRM-free version of those games comes with several folders and each episode has to be installed separately... I just hate it when games do things differently.
Of course it's all due to Steam starting as a store for Valve games, then evolving into adding AAA games from other publishers, then AA, then A, then Z-... :-P Each time they open themselves up a bit, they lose their grip on how authors use Steam.

[r6d2]

Publicado originalmente por Nao 尚:

How do you know that for sure..? to me, it seemed that if a parent was set to '0', it didn't belong in a sub...

You can check it for yourself by several means. Try these:

• Go to your Licenses and product key activations page. Every name you see in there is the SubID name, not the AppID name, even though in some cases it's the same name.
  
• Now go to your Userdata page. After "rgOwnedPackages" you'll find the list of SubIDs associated to your account. You will also find the list of included AppIDs on those SubIDs after "rgOwnedApps".
  
• You can also login to SteamDB.info and, if using their browser extension, your packages and games will be highlighted. You'll find a "You own this package" green sign on every package you own.
  
• If you still have doubts, open the Steam console and run the "licenses_print" command.

Publicado originalmente por Nao 尚:

Now, regarding the database, [...] I'm not sure where they got this from, really. But it makes it harder for me to just say, "okay, I'll stick to scraping"... What do you think?

I think you'd better stand on the shoulder of giants and not trying to build everything from scratch.

Both Depressurizer and SteamDB are great sources of info and will let you concentrate on the real core of the new site, which is the trading mechanics and not every significant and insignificant little detail of 100% of the apps since day 1. Remember the Pareto principle :-)

[devotee]

Publicado originalmente por Nao 尚:

However, like Milouze I love the store tags (again, I'd rather use them as "source").

Yeah, 'source' is a better word to represent them indeed.
I also tried to consider having a 'steam/origin/uplay/gog..' tag instead of the mess that is on Barter, but I changed my mind quickly after I realized that many games on the alternative stores aren't on Steam, so I can't really force them all to have a Steam ID.

Regarding stores/bundle sites, I wouldn't remove those which are still working (Lazy Guys, for example). It's true that we don't know how long they will be around, but in the meantime I think we should keep them. And even if they do disappear in two years, that doesn't mean that we won't still have steam keys floating around from them :)

I don't know how much trade is done on barter with non-Steam games, but I would say it's a pretty low number. I think people on barter mainly focus on Steam games, and most of us don't care/don't want games from Origin or UPLAY. I personally hate UPLAY[i.imgur.com]. Limited stuff, DRM everywhere, buggy clients,... GOG is a total different story, but still not worth the effort of adding it to your new site.

Publicado originalmente por Nao 尚:

but I also want the freedom to add professional effects to a website. And that implies external CSS and JS... (I'll consider switching jQuery to Zepto for better mobile performance, but I'm not sure it'll help much.)

That's true, I really miss on barter features like auto search when typing, dynamic elements, Ajax methods to do everything in a single step/page instead of having to constantly reload it with every change in the offer form, etc.

Publicado originalmente por Nao 尚:

Match speed, however, is atrocious... I can't think of a single reason why matching user tradables & wishlists would always take more than 30 seconds. We're talking about a table intersection, here... It should be really, really fast. Under a second.

The match system works fast when it comes to user-to-user match. If I do a site-wide match (who owns a game I want and wants something from me, for example) it does take forever.

Publicado originalmente por Nao 尚:

A nice but controversial feature would be to be able to store our keys/gift links in our accounts.

It's not controversial, it's planned. Storing both links and keys. For yourself or for retrade.
The main idea is that you get to choose how you want to trade it...

The process would be similar to what Lastpass and other pasword managers do, you store the heavily-encrypted information on the site's database and the decrypting process is done on the user's computer using his own password/passphrase (which is never sent to the site).

I'm not sure about that, though.
If you lose your passphrase, then the codes are lost forever.
I'd rather not encrypt anything, and just encourage people to only storage their bundle stuff.

I'm not sure if you should do that without some kind of extra layer of security. If you simply store the information in the database that can lead to several problems:

- If someone hacks the site, he would have access to the raw keys and gift links
- If someone is 100% sure a key is working, he trades it but it doesn't work and it was in the database, they might get suspicious and maybe even claim that the site admin or the people who have access to the database stole his steam key. Even if he's wrong, or the problem comes from the original source of the key (it wouldn't be the first time that a bundle site gives someone a duplicate/invalid key).

I'd rather have to remember a passphrase I can store somewhere secure (even have a printed, on paper list of "one-time-passwords" in case of emergency to recover access). If the site warns me to keep it in a safe place and protect it like my own life, and I don't, it's my fault. I can't blame the site owners for my mistakes. But if something "strange" happens with a key/gift link, etc. and the people involved in the site can claim they don't (can't) have access to that data, then there's no need for suspicions or accusations.

You have dealt with a large number barter users (you and me both), and you know that some of them would go all crazy if they thought something "fishy" happened to their precious Woodle Tree Adventures Steam key

[devotee]

Sorry, I forgot to ask one thing(and I might need to apologize for asking this )... How are you going to deal with free given away games on your site? A lot was discussed in this thread.

I guess the better way is to allow people to tag a game as "given away to general public" (like barter does), but also allow to add additional information, like the give away url (or link to some proof), the number of keys given away (if known), and the date of the give away. Like you said with the bundles feature (comparing how barter is faster adding new bundles than ITAD), if we're fast enough that would allow people to try to get their own key on the give away. On another hand, knowing the date(s) of the give away(s) would also allow to know if it was a rare giveaway, a common given away game (Woodle Tree Adventures, The Deer and such come to mind), and if a user should expect a new give away anytime soon (if is a usually given away game, he might want to wait instead of trading for it).

That's the most basic approach. Other exotic features like temporal banning of trades for a currently given away game, or allow to report false user tags,... would be nice but not required. I have the feeling that some people try to abuse[barter.vg] that barter's feature, maybe trying to deceive a trader, tagging a game as given away and trying to make him think that the game was given away and it's value has now decreased.

EDIT: Forget about the "date of the give away", that should be the timestamp when the user added the "given away" tag :P But it's useful to store every tag's timestamp in order to know how frequently a game is given away and if it was given away recently.

[theemu]

Publicado originalmente por Nao 尚:

How do you know that for sure..? to me, it seemed that if a parent was set to '0', it didn't belong in a sub...

There is Steam Sub package number 0: https://steamdb.info/sub/0/

The only way to "own" Steam App is to "own" Sub(scription) package that contains that App.

Example: there are fully removed games such as vanilla version of Jagged Alliance 2: https://steamdb.info/app/1620/subs/
As you can see at SteamDB, App 1620 exists, but it does not belong to any Sub package, so it is impossible for anyone to "own" it. On Barter you can find most of these removed Apps. On new site it would be nice feature if such Apps could be flagged so that they are impossible to add to tradables/wishlist.

[r6d2]

Publicado originalmente por theemu:

For example, it seems Barter is set to ignore Apps that have value for Price set as "No Price" (look via SteamDB.)

My guess is that this is not a Barter setting, but some kind of syncing thingy, since the admin told me once that he was not able to get the price of every app, otherwise he would always have them available on the site.

PS: your quoting on the previous post seems to imply that I said what actually was said by Nao ;)

[r6d2]

Publicado originalmente por theemu:

Example: there are fully removed games such as vanilla version of Jagged Alliance 2: https://steamdb.info/app/1620/subs/
As you can see at SteamDB, App 1620 exists, but it does not belong to any Sub package, so it is impossible for anyone to "own" it.

The weird thing is that you can still go to the store and buy it, so my guess is that in this case SteamDB is just not updated about which sub this app currently belongs to.

[r6d2]

Publicado originalmente por theemu:

On Barter you can find lots of these removed Apps. On new site it would be nice feature if such Apps could be flagged so that they are impossible to add to tradables/wishlist.

I agree this would be nice indeed.

Having the option of flagging or banning free apps from being offered and traded would be a plus too, IMHO.

SteamDB maintains a list of such apps here.[steamdb.info] The catch is that the numbers listed there are SubIDs, not AppIDs, so some form of mapping would need to be done.

Also, some apps are known to become F2P soon and still some people is still trying to trade them, like Faeria[barter.vg], for example, which is highly demanded on Barter.

[theemu]

Publicado originalmente por r6d2:

Publicado originalmente por theemu:

Example: there are fully removed games such as vanilla version of Jagged Alliance 2: https://steamdb.info/app/1620/subs/
As you can see at SteamDB, App 1620 exists, but it does not belong to any Sub package, so it is impossible for anyone to "own" it.

The weird thing is that you can still go to the store and buy it, so my guess is that in this case SteamDB is just not updated about which sub this app currently belongs to.

Not exactly correct.
It's true that Steam Store page for Jagged Alliance 2 Gold is under AppID number 1620. But when you add it to shopping cart you can see that what you actually will get is Sub package number 94: https://steamdb.info/sub/94/ which contains Apps 12370 & 12380. There is no way around it, it's not possible for anyone to own AppID 1620.

Similar case is for example Gold Edition and Black Edition of one Painkiller game. Steam Store page is under AppID 3200: https://steamdb.info/app/3200/ But when you add to shopping cart currently, you are actually buying AppID 39530: https://steamdb.info/app/39530/ (the currently sold Black Edition version of the game).
AppID 3200 is the old Gold Edition version that is no longer sold anywhere. It is very rare to find nowadays, only subs 251 and 249 contain it: https://steamdb.info/sub/251/ https://steamdb.info/sub/249/ Sub 249 is somekind of Valve test sub, and 251 was sold only via Steam Store, so it's possible to get as Steam Gift only.

Steam Store pages are not totally reliable source of information about Apps & Subs when it comes to games that have many different versions. Store pages are also subject to region restriction stuff. SteamAPI is better source, and SteamDB is best source of information I know thanks to the fact they have the change data archived for each App and Sub.

[r6d2]

Publicado originalmente por theemu:

It's true that Steam Store page for Jagged Alliance 2 Gold is under AppID number 1620. But when you add it to shopping cart you can see that what you actually will get is Sub package number 94: https://steamdb.info/sub/94/ which contains Apps 12370 & 12380. There is no way around it, it's not possible for anyone to own AppID 1620.

Oh, I get it now. Thanks for clarifying!

[theemu]

Publicado originalmente por r6d2:

Publicado originalmente por theemu:

For example, it seems Barter is set to ignore Apps that have value for Price set as "No Price" (look via SteamDB.)

My guess is that this is not a Barter setting, but some kind of syncing thingy, since the admin told me once that he was not able to get the price of every app, otherwise he would always have them available on the site.

For example some games & DLCs are only sold&activated by CD Keys, like bundles. Those Apps never get synced to Barter automatically because they don't have Steam Store page even. I don't know how important getting&storing the Steam store price really is anyway. ITAD already does it and I think traders count the actual bundle cost value or go by the current resell price.

[Nao 尚]

Gosh, plenty of posts... I can't keep up!!

Publicado originalmente por r6d2:

I don't get why you are not inclined to use the export feature. I dump the whole Barter Steam database every week without any hassle (games only, not packages) including the only primary key that counts, the AppID.

Export = user has to go to barter.vg, use that feature, and paste it into the new site.
My system = user logs into new site, site notices he's a member of Barter, and imports their tradables automatically...

BTW, currently the Steam API returns 28K entries. Barter is currently populated with a few less than 23K. Of that total, only about 8K are really current, and only about 2K get updates on their Have/Want/Own quantities with some frequency.

I'm planning to keep everything into the database, just 'disabled' (or rarely updated) for scrape updates if the games aren't too important (e.g. old release, decommissioned, maybe DLC, etc.)
BTW, on Barter's list of games, there are 'only' about 16500 entries for Steam games, and just a handful for non-Steam. (Mostly Desura.)

I really don't see myself importing a list of Desura games... Some people have those in their tradables, I'd be inclined to just ignore the entries. I can't waste time on a dead platform..!

BTW, using name for lookup purposes will only get you into trouble. I'll mention just a few classes of problems:

Several AppIDs have identical or very similar names. Sometimes these names refer to the same object, other times they don't.

Yes, I'm aware there are a few games made by different people that have the same name. But what should I do then..? Store the barter ID in the database as well..? I can do that, but it's boring. Plus, it means everyone gets their 'bad' entries imported. ^^

Even Type+Name is not a valid primary key (type being the SteamDB/Depressurizer reported Type such as Game, DLC, Video, etc).

Not planning to have a key on type+name... Currently I'm using a fulltext index (maybe that's overkill...) on the name field, at least the search system will work better than on barter... ^^

My algorithm is very straightforward to calculate, and I have no problems in sharing it with you.

Whenever you want :)

Of course it exports tags. It does even better that that, it also exports if something is a key, a link or a gift. Check the link I pointed on my previous post. It documents every little detail.

The admin probably added that feature later (that select box), but it's still not something I understand... Why rely on text imports when I can get the exact same information from scraping..?

[theemu]

Publicado originalmente por Nao 尚:

Gosh, plenty of posts... I can't keep up!!

Sorry! Now on I'll try to take part in discussion, instead of starting a lot of new topics.

Publicado originalmente por Nao 尚:

I really don't see myself importing a list of Desura games... Some people have those in their tradables, I'd be inclined to just ignore the entries. I can't waste time on a dead platform..!

My suggestion for adding non-Steam games/software to the database: Just leave it up to the users to add them. Eventually database would get filled with all Origin/Uplay/itch.io etc games that people have for trade/wishlist.
New database entries should be approved by some trusted users of course, but that system is not hard to set up at all. I can foresee users who would approve new entries to the database, and "sheriffing" the site for duplicates and spam.

Several AppIDs have identical or very similar names. Sometimes these names refer to the same object, other times they don't.

Publicado originalmente por Nao 尚:

Yes, I'm aware there are a few games made by different people that have the same name. But what should I do then..? Store the barter ID in the database as well..? I can do that, but it's boring. Plus, it means everyone gets their 'bad' entries imported. ^^

I'd say that at this point, the entries should just have the official title from Steam + the similar title warning texts added at Barter?

The 'similar title warning text' system on Barter needs to be improved also. On Barter those warning texts are not visible on the site globally + the space for typing the note is too limited. Sometimes being able to add an URL would be easier than typing a short note for example.

New site could have 'are you sure you have chosen right item?' section. When user has added commonly known wrong version of the game on their tradables, the site would show warning and suggestion to change it to the correct version.

[r6d2]

Publicado originalmente por Nao 尚:

Export = user has to go to barter.vg, use that feature, and paste it into the new site.
My system = user logs into new site, site notices he's a member of Barter, and imports their tradables automatically...

That would be a much better UX, of course, but you were complaining about parsing the HTML, so I suggested to use something that's already available. ;)

Barter is currently populated with a few less than 23K.

BTW, on Barter's list of games, there are 'only' about 16500 entries for Steam games

May I ask how are you getting this number? When I count I get 23K distinct AppIDs for Steam items on Barter.