Building Achievement Tools, Any Feedback?

[Nerdburglars]

I have been working on creating some tools that will benefit achievement hunting and would love some feedback on it. The Steam API has very limited support for achievements, but i have been able to build up some work arounds to get this working. I have created a system similar to any of the xbox achievement sites where people can submit guides and mark off completed achievements etc. https://nerdburglars.net/steamachievementlists .

I have also setup a Steam leaderboard allowing for users to get their Steam profile up on the leaderboard. The next challenge i face is linking the users to the achievements. Steam does not return a list of achievements based on a user. However, it does return a list of achievements for a particular game and user. What i was thinking of doing was adding a sync button to the achievement list pages. This would get the achievements you have unlocked and would automatically mark the achievements as unlocked. This could then be used to add acheivements to the leaderboard.

I have written code to get all achievements by getting a list of all games a user has and then checking the achievement progress for each game. Problem is this takes 3 or 4 minutes to complete for users who have 100s of games. There isnt enough hours in the day to keep an up to date value for all steam users.

Anyone have any feedback or suggestions on how i can improve this system to make it a better tool for achievement hunters?

[Nerdburglars]

I suppose its similar, but I want to be able to link the achievements directly with the Steam API. At least this way you can mark your progression on the achievement guide. You can pop over to the list for a particular game, see which achievements are remaining and the list you see will be a guide to get the achievements that you have left to obtain

[luchaos]

hey there!
developer of completionist.me here. i like what you put together so far!

as for those achievement stats - you found the only way that can be done.

please know that you're about to enter hell's gates when dealing with steam's apis - here are some suggestions and warnings:

- use the steam web api, not the deprecated community data xml api (except if you need unlock timestamps, those are not in the web api)
- to comply to steam web api terms of use only fetch data for users that logged in through steam's openid (if you care) users like privacy
- have a queue system in place on the server side that fetches data for users, games and users' game stats asynchronously so you can manage the 100k requests/day rate limiting. you won't hit that anytime soon but there will be bottlenecks when users with 10k games and 30k achievements want their profile added. also, community data's rate limiting is even tighter.
- the worst part: there are more edge cases in steam's apis than "Tales of Maj'Eyal " has achievements

i won't even go into detail there just remember to fight carefully when you're about encounter these dragons: expiring licenses (demos, free-to-play, free weekend games, family shared games), games with custom stats pages that won't report timestamps through community data (Alien Swarm, Payday, ...), improper encoding in game and achievement names, completely incorrect game names, games with all achievements removed, achievement ids not matching up without prior normalization, community data's achievement unlock timestamps disappearing, steamcommunity being down for unscheduled maintenance, and many many more...

if you dig solving problems i can absolutely recommend tackling this beast - it can be actually very fun. bear in mind that here the cakewalk is a lie.

keep it up! i added your site to my list of tools and trackers ;)

cheers

[Nerdburglars]

Thanks for the feedback.

Using the main JSON based API at the moment. I had considered the XML one for the timestamps, but people are saying that it only has a timestamp for some achievements that were added after a certain date. Ill leave this on the nice to have list for the moment.

When you mention the open Id, is this in reference to linking a user profile to the achievements? So if a user wants to sync the achievements they have they have to login with the openID system rather than having to do something else that is less secure?

As for fetching the user data, i have written everything using objects. I then cache the object as a json file to save making a crazy amount of requests to the API. One user hit an 8mb cache file just for the achievement data, which is why i ditched the idea of getting everything automatically. The server has a 2TB drive, but that will quickly run out when each user has 15mb+ of cache for all the profile data just for cache! Rate limits on APIs have been kicking my ass for the past few years so ive done all i can to make a call to the API only when i have to.

I appreciate the feedback though. Im glad you shre the same feelings i have gone through while building what i have done so far. I started out with "why hasnt anyone used this API to build much stuff so far". Now i understand!! :P Ive gone quite far with it now though, be a shame to give it up. Feedback like this keeps me motivated :D

[luchaos]

awesome - glad to hear that! let's say those apis are ... not the best :D

about steam's open id login - that's about the formalities that steam has in its terms of use: https://steamcommunity.com/dev/apiterms

as for the implementation details to comsume api data - it does not change anything at all. as soon as you have that api key of yours you can fetch all the available data regardless of the user's current steam login state on your site. there's no actual oauth with token exchange, grants and scopes behind any of this.

i used to drive completionst.me from cache, too in its early stages (actually still do for some parts for convenience and performance enhancement).
reducing api calls to the absolute viable minimum was hard but very important - very good that you have that in place.
aggregating and querying that data still requires the data to be persisted in a (relational) database - those still do the best job for storing large amounts of data in the best optimized ways. cache invalidation may happen much earlier that way, too, which in turn frees up resources.

exactly, never give up. you seem to have worked around psn's non-existing public api, so steam's may or may not be worse - still, it's just another api.