On 2015.05.25 the beta branch is now public!

On 2015.05.15, we launched a private beta branch with access to a newly implemented workshop. This post will serve two purposes which will give people a heads up on what to expect, but also to give instructions on how it is going to develop over time. This is going to be a time of transition, so I'll do my best to explain the situation and the goals:

The original workshop implementation was launched with the original release of the HD Edition just over two years ago. Back then Steam Workshop implementation was quite different than the features available now. But over the years we've implemented Steam Workshop for Age of Mythology and Rise of Nations, and in the process the back-end has changed signficantly and our understanding of how things worked has also increased.

The way the current workshop works is when an author makes a mod, they modify a bunch of files, build a list of files inside the game, and then submit them to Steam. The game zips all the files into a single file and uploads it to Steam. And it always publishes it as a new workshop item.

When people subscribe to them, and run the game, the game will explicitly download the file into the mods directory and extract the content over top of the root files. Because it was interfering with original game files by replacing them, Steam couldn't possibly know how to deal with these files when a user unsubscribed to them due to the custom nature of how subscriptions were handled in the game. This is why people had to verify their cache in order to properly clean up.

Also it was impossible to publish updates to an existing mod, so every time an author made changes to a mod, they would have to publish it as a new item, and lose all comments, ratings, and usage stats.

There was also no possible way to manage mods in a dynamic way. No way to prioritize them, enable or disable them, stuff like that.

Another major problem is that the game has a lot of large archive files that contain large numbers of smaller data files. These are DRS files, so the problem is if someone wants to modify a DRS file, only one mod at a time can work on this.

So the goal is to solve all of these problems.

SteamRemote is a Steam server which is responsible for cloud storage of files. Cloud storage has a total maximum size of 1000 files and 1 billion bytes, whichever hits first. This includes save files, replays, etc., and published mods.

I believe that is why the decision was made to make mods into zip files -- so each mod would be considered 1 file. Otherwise, it would be very easy to hit that 1000 file total limit. Modders quickly found themselves hitting this limit anyways and being unable to publish mods. Also mods had a total limit of 100MB per mod, so each mod was also limited in size.

These were fairly serious constraints and both Age of Empires and Age of Mythology are subjected to these constraints. I believe when Valve implemented Steam Workshop, they never really intended for it to really have to co-exist with steam cloud data, and they probably never expected how popular workshop content could be.

SteamUGC (User Generated Content) was introduced on June 10, 2014, less than one year ago. This was Valve's major restructuring on how workshop content could work, and this is what we embraced the moment it came out to implement inside Rise of Nations, which shipped only two days later. It is here we learned just how superior this new implementation was, and worked on it for a couple months before we finished it and released it as a post-launch patch.

File sizes and limits are completely gone, so authors can publish everything they want. There was also the ability to publish a "directory" where it would publish everything inside that directory, and manage the compression, and "delta transactions" of updates automatically. So say you make a 600MB mod, and upload it. Steam will automatically compress it and upload only the compressed amount (lets say 200MB). Then later, when you update your mod and change a single file -- Steam is smart enough to know the differences and only upload the difference which may only be 5MB. Finally, the game is able to talk to Steam and request real-time progress information on uploads and downloads.

Valve also made a number of other improvements under the hood that made development a much easier than in SteamRemote. Unfortunately the differences were signfiicant enough that a total reimplementation was necessary to solve the problems inside Age of Empires.


If you've seen the implementation of Myth or Rise, you'll get a sense for the evolution over time. Both of those games have a built-in mod manager that allows you to view, prioritize, and enable/disable any mod, as well as the ability to publish and update mods automatically. The beta contains a similar implementation, but we would consider it a step forward.

Startup Validation
When you first start up the game, we initialize the Mod Manager. What it does is registers any local mods (applicable only to authors), then enumerates anything that you've published or subscribed to. And it will display everything in real time before the main menu. For most users this will happen so fast, you won't even get a chance to read it. But if you subscribe to a lot of mods and mods need updating, it'll take longer. You can choose to dismiss it automatically or wait for you to close it.

How mods live
Mods are no longer zip files. Most mods don't even live inside the game. Steam automatically manages subscriptions outside of the game and stores them in deeply buried Steam directories. The game no longer actively downloads any files. We let Steam do everything it can such as subscriptions, unsubscribe, and auto-updates. The only mods that actually live in the game's mods\ directory are mods that authors make for testing and publishing purposes. Once published, the mod will still live in the mods directory for future editing and updates.

How mod files are loaded in game
When a file is loaded in the game, such as a tree or a terrain texture, it will look for the top-most priority mod that is enabled, working it's way down the mods until it either finds a match, or uses the original asset. For this game, we've made virtually every file moddable.

New data structure
In order to support maximum ability to mod, we have completely eliminated DRS files which used to be containers for several other files. In fact, we've reorganized the data structures to better support modding and ease of translation. It's pretty detailed, but the information of remapping is located here: https://docs.google.com/spreadsheets/d/1hlCcJzNQY14rruNzAFnEPncrKy6yfIPdAbp5mZYP2zM/edit#gid=0


How to approach using the Mod Manager Beta
For now it is a private-invite-only beta, but we will open it up to the public shortly before making a global switch.

The problem we face right now is a transition from what we're calling "legacy" mods which is basically everything in the workshop right now. There are 12,000 mods in Age of Empires right now. What we really want to do is move all the legacy mods to the new system, but when we make this switch, then everyone running the current live branch won't be able to load them because the old game doesn't understand how to use the new data structures or how to use the mods because everything has fundamentally changed for the better.

So this first version of the beta won't allow transfers from legacy to new. But you can take a legacy mod, and upgrade it. In the process it'll convert it to the new format, and publish it as a new mod. This is so people can test that old mods work under the new system. Once we prove out the system, we'll flip a switch and there will be a window of time where legacy mods will move to the new system and no longer be available to the old system.

Uh, so what does that mean?
Well, best to use an example. Say you have a subscription to one of Ozhara's terrain texture mods, and you use it all the time. Right now if Ozhara upgrades it to the new system, it'll exist in both locations and work in both locations. But all the comments, ratings, and usage stats won't transfer over to the new system, and we want to preserve that.

So say in a week, we flip the switch and he formally moves the mod to the new system, if you are still subscribed to it, it'll actually still work because it's already been applied to your game data files. But if you don't have it subscribed and installed, you won't be able to use it until we make the patch live to the world.

What will happen is when the upgrade happens, the mod will cease to exist in the legacy system, and transfer to the new system, preserving comments/ratings/stats. When the patch comes out, your subscription will still work and you won't even really know anything has changed.

Notes for Authors with access to Beta
What you want to do is subscribe to some of your mods outside of the game and switch to the beta branch. Then when the game starts, it'll register the subscriptions and you'll have access to them. When you enter the Mod Manager, you'll see the mods in a list. All the mods should show up as legacy mods and be forcibly disabled and in red.

But, an upgrade button will be available for mods that you published. If you didn't publish it, you can't upgrade someone else's mod. If you hit the upgrade button, it will attempt to transfer it to new system. If it succeeds, it'll automatically publish it (as a new item, and forcing it to be visible to friends only) -- because we are just testing these mods.

There are only two types of data that will not transfer. That's DRS files and text files. If your legacy mod contains DRS files, you''ll need to break it up and manually create your mod with only the files that you actually modded, because DRS files are no more and everything exists as loose individual files.

Text files is a more complicated issue. We have rewritten the text system and merged all the files into a single text file (unified DLC and game). We've also changed it so you can now switch languages on the fly from the options screen. But because the game is 15 years old, the text encoding was rather ancient and doesn't really work on modern systems. So everything is now UTF-8, so you can see all languages now. Also the intention for text files is to "add" or "modify" some text -- only things applicable to your mod. So text is additive in nature, so you only need to specify what you care about rather than replacing the whole thing.

If you want to modify the text file in your mod, you'll need to copy the following file into the same directory in your mod:
Age2HD/resources/en/strings/key-value/key-value-modded-strings-utf8.txt
into:
Age2HD/mods/MYMODNAME/resources/en/strings/key-value/key-value-modded-strings-utf8.txt

Then it's business as usual, just add or replace strings from the original text file. English is treated like a global fallback language (like the former language-base.txt), and if you want to translate your mod in other languages, just do the same thing in any or all languages (replacing en with fr or de, etc.)

When an upgrade is attempted, it takes your legacy mod, extracts the zip folder into your mods\ directory and remaps the files based on this link: https://docs.google.com/spreadsheets/d/1hlCcJzNQY14rruNzAFnEPncrKy6yfIPdAbp5mZYP2zM/edit#gid=0

If it fails due to containing text or DRS files, it'll still remap everything else, but failed files will be left alone in their original place in a subdirectory called _extractedfiles\. Then a windows explorer opens up to that directory automatically and you can manually fix up the remaining issues. When this is the case, you'll need to publish the mod manually.

If it succeeds, publishing will happen automatically.

I know it's a lot, but I'm happy to answer questions. If you want access to the beta, post here and try to friend me in Steam.

Known Issues
I'll be listing known issues in future posts and I continue to patch this beta :)