converting saves from earlier versions

[EmotionallyBroken]

I am looking for any information on how to update saves to be compatible with newer versions of the game.
Ive read at some point a tree mechanic was changed and that has to be adjusted at least.
Ive noticed saves are broken down already into separate directories and files, so this may be easier than most other games (Thank you Tarn and company :)

this is basically the only information I have at the moment, and anything else would be welcome in this effort, perhaps Ill write a guide. (or someone else can)
That is information other than "You cant do that, I think is impossible" As not only is saying something is impossible in an entirely virtual environment very silly, which isnt constrained by the physical limitations of hardware simulating it, its well... simply not the topic of this thread. which is information about how to update saves to be compatible with the newer versions of the game :)

A submeasure is to use quickfort (from my understanding this copies the fort design only, and nothing in the map/world itself other than the fort) not quite what im looking for, but this might be enough for some people.

Anyone who has save or other editing knowledge that might help, please contribute.

[rome of oxtrot]

so far all version of v50 are upward compatible: all v50 saves can be loaded in 50.14. v51 (beta) saves cannot be loaded in any version of v50. i've heard reports that 50.14 saves cannot be loaded in v51 but i have not verified this. i don't have a matrix for downward compatibility as i don't offhand know where our matrix of save versions to game versions is, so i'd probably have to go through the reverse engineering repository to pull these out for each released version

the wiki has the details on compatibility for version prior to v50; see https://dwarffortresswiki.org/index.php/Save_compatibility

[rome of oxtrot]

i dug through our reverse engineering logs and reconstructed the matrix for the releases since December 2022. i'm not going to do every DF release as there are too many of them and pulling the save version and minimum load versions out of them is more work than this discussion is worth

as a reminder: each version can load any save whose save version that is at least as large as its minimum load version and not larger than its own save version. the save version written into a save is the save version of the version of the game that is writing the save. it does not matter what version was used to create the game originally

50.14 and 50.13 are save version 2082. 50.05 through 50.12 were 2081, and 50.01 through 50.04 were 2080. the minimum load version for all v50 releases is 2047, which does not correspond to any known release of Dwarf Fortress and was presumably a development build that never escaped from bay12's proving grounds

51.01-beta22 and 51.01-beta21 are both save version 3005 with a minimum load version of 2047. i don't have data conveniently available for any other of the 51.01 beta builds. 51.01-beta22 is the current beta build

47.05 was save version 1716 with a minimum load version of 1443

toady generally does not indicate why he decides to bump the save version. in some cases we have determined what the reason was through analysis of the deserializing code in the the game (which will need to behave differently depending on the save version being loaded), but we only do this when it piques our interest to do so, and i don't think anyone has done it for the two version bumps in the v50 era (50.04→50.05 and 50.12→50.13)

[rome of oxtrot]

as to "updating an old save to load in a newer version", don't even bother: unless your name is Tarn Adams, you don't have anywhere near enough information to do this, and the fact that Tarn didn't put it into v50.01 leads me to think that he didn't either

to be able to do this, you'd have to extract the deserializers from an appropriate old version of DF, the serializers from the desired new version, and a transformer to translate the game data structures. for v47 to v50, this would be exceedingly difficult, due to the rather substantial changes in how civzones are implemented. not to mention that we haven't even identified all of the deserializers in 47.05 (we probably have about 60% of them tagged) or all the serializers in 50.14 (probably about 70%). and information on the meaning of internal game structures is still fragmentary, especially in older versions

Tarn has historically done a very good job of preserving save compatibility through upgrades. there's only been one save compatibility break from July 2014 to current, at the December 2022 release of v50.01. v47.05 (released January 2021) can load a save from any version from v40.03 (released July 2014) to v47.05. i think it's a safe bet that Tarn is doing everything he can to maintain upward save compatibility going forward, and i wouldn't be all that terribly surprised if v100, when it's ultimately released, will be able to load v50 saves

[EmotionallyBroken]

Ummm....
Just how many compatibility breaks have there been since 34?

Also, your motivation and drive, and your clarity through ,thoroughness and detail is outstanding, and I very much appreciate it :)

[rome of oxtrot]

Originally posted by EmotionallyBroken:

Ummm....
Just how many compatibility breaks have there been since 34?

five: 34.02, 40.01, 40.02, 40.03, and 50.01.

[EmotionallyBroken]

Originally posted by rome of oxtrot:

Originally posted by EmotionallyBroken:

Ummm....
Just how many compatibility breaks have there been since 34?

five: 34.02, 40.01, 40.02, 40.03, and 50.01.

Thank you.
Can you share from memory, or link to notes on what causes these incompatibilities? Looking at the wiki I am only finding a simple note that the saves are not compatible with newer versions and no further elaboration, but I may be missing details or links elsewhere.

As a better half measure than merely quickfort, I am wondering and about to try copying old save world data or geology over a newer save. I am just getting familiar with the file lay out of saves and not sure how separate these portions are. This idea is of course less exact than a full update of the save correcting only the minor changes needed.

[lethosor]

In the cases of 0.34.01, 0.40.01, and 50.01, maintaining save compatibility was too hard due to the number of things that changed.

Not sure where you were looking on the wiki, but https://dwarffortresswiki.org/index.php/Version_history mentions save corruption in the cases of 0.34.02, 0.40.02, and 0.40.03. If you follow the links in that table, you'll find pages with the release notes from each version, e.g. https://dwarffortresswiki.org/index.php/v0.34:Release_information/0.34.02. Those are also on the old devlog: https://www.bay12games.com/dwarves/dev_2012.html for 0.34.02, https://www.bay12games.com/dwarves/dev_2014.html for 0.40.xx.

[EmotionallyBroken]

Originally posted by lethosor:

In the cases of 0.34.01, 0.40.01, and 50.01, maintaining save compatibility was too hard due to the number of things that changed.

rather vague, but I cant say inaccurate in the slighest. Thank you for the links to the slightly deeper information.

through some expirimentation, Ive learned saves will load if all the files are replaced or even deleted other than the world.sav, which appears to be the largest , and contains most of the save information. I after also found this https://dwarffortresswiki.org/index.php/User:Andux/Format_research/WORLD.SAV
I have the feeling you may already know all this :)
maybe worldsav is the created state, and the other files are changes to the starting state, but thats pure theory.

the file appears to not be in text format as some other files are, and perhaps the next step is to try and read and edit it with a dissembler or hex editor to see if anything recognizible can be seen and adjusted to the updated state. Another thought it maybe DFHack might have some tools that might be useful in that, Im not very familiar with it. Have either of you ever used it?

Is there a clear and full breakdown of how to fully and clearly see wahts in saves somewhere? That would be the easiest way to see whats going on and be able to figure out how to update saves. For all I know there is something equal to total wars save parser and I am totally unaware of it. :)

[rome of oxtrot]

We know exactly what format DF save files are in: they're raw serialized binary, optionally compressed with zlib. The problem is that the raw serialization format is extremely fiddly to parse: the file contains no structural indicators at all, and is just a stream of bytes which are interpreted by the myriad "read_file" methods within DF's code itself. There are at least 732 deserializing methods in DF (at least that's how many we know of in version 50.13, which is one of our better analyzed versions), so this is an extremely nontrivial exercise.

Developing a tool to read (or write) DF save files requires extensive reverse engineering which would have to be redone for each new save version. In general, we find it easier to let DF do the read, inspect the in-memory representation, and then let DF do the write, than to write our own readers and writers.

Deleting the unit and art offload files will not prevent the save from loading but it badly damages your save and will probably cause the game to crash eventually. (We also know exactly what those files are; your "pure theory" is not accurate.)

The two people you are talking to in this thread are two of DFHack's lead developers.

[lethosor]

Not much to add from my end. I do recall trees being one reason that 0.40.01 broke compatibility (going from single-tile to multi-tile trees was nontrivial). I believe I heard that tree graphics were a compatibility challenge in 50.01, but I don't know that there is a deep-dive from Toady into that.

DF does have _some_ logic to recreate .dat files that go missing or get corrupted. Specifically, corrupted unit files used to be a common-enough cause of crashes that Toady added a feature that would attempt to rebuilt them from scratch if corruption was detected, as a last-ditch effort instead of refusing to load the save at all. Obviously this results in some data loss, because the data stored in those files is not stored elsewhere, but it only affects the units stored in the replaced files. As far as I know, this does not apply to the other .dat files (e.g. art images) - not sure that deleting those is safe. Also, replacing any .dat files with _valid_ files from another save would probably cause strange behavior at least, if not crashes.

[EmotionallyBroken]

Originally posted by rome of oxtrot:

The two people you are talking to in this thread are two of DFHack's lead developers.

Well thats just fantastic! haha
This makes talking about things that are already as it, and talking about new possibilities to build off of that work much easier :)

The unit data I thought might be about living creatures and possible non living creatures everything that is physically in the world that is not the terrain.
Trying to think of a way to build and copy old data into the new data format of a newer save , to if not get something partly working, at least learn something about it, I had a thought while seeing "unit" files that the world sav may be the terrain of the world , and unit data being all the living creatures, copied onto a completely different terrain world would would cause a lot of units loading at.... wrong elevations :) ..... and... a lot of blood spattering , and other unforeseen effects.

I did not do so much as unpause it, and its quite possible, if not highly likely that game would immediately crash, if not .... exhibit some strange or potentially amusing behavior., it was really only testing for the sake of learning whats possible and how things are set up and work, and isolating what parts of the save are what.
I found https://dffd.bay12games.com/file.php?id=14727 here, which looks like it can decompress or otherwise somehow extract save data, its versions are limited but in a wide scale and that may help me learn what the art and unit files are. Though it sounds like you already know :)

Originally posted by rome of oxtrot:

we find it easier to let DF do the read, inspect the in-memory representation, and then let DF do the write, than to write our own readers and writers.

That said, perhaps the easiest way particularly if the full state of game is known in ram while running, is to alter the game state in ram on the older versions of the game , "prebreak" , pause the game (probably makes thing easier) and alter the info in the ram into the state it would be in in the next compatibility break phase, copy that ram data , into the ram data of a process of the game running in that later phase, and have the newer post break version save it !
(This of course would require some generating for some things ,like if trees go from single to multi blocks, which could be perhaps done with some generating code :) maybe write a script to delete old trees, after saving their locations, and generate the root/trunk in the same place, the games already present tree generating code would be optimal, and match better the way it generates things of course .


Have you already tried or thought of this kind of approach? It would certainly make a huge difference is Tarn or Zach were to help. Have you tried asking Tarn and Zach (or I heard they have a new third helper?... putnam??? ) for help with this? Am I even thinking right in assuming youve tried to make save updating at all, or is that just me? I think my brain energy is going down now so if Im getting less concise and cohearent, please forgive me. Im going to just pause now.

Anything is possible with the virtual, with a wide range of circumstances and ease/difficulty.

[rome of oxtrot]

While I could respond to most of the above, I'd rather not do that in this forum; Steam discussions are just about the worst possible place to discuss in-depth technical matters. Feel free to join the DFHack development discord if you wish to discuss any of this in more depth.

[lethosor]

Originally posted by EmotionallyBroken:

Originally posted by rome of oxtrot:

we find it easier to let DF do the read, inspect the in-memory representation, and then let DF do the write, than to write our own readers and writers.

That said, perhaps the easiest way particularly if the full state of game is known in ram while running, is to alter the game state in ram on the older versions of the game , "prebreak" , pause the game (probably makes thing easier) and alter the info in the ram into the state it would be in in the next compatibility break phase, copy that ram data , into the ram data of a process of the game running in that later phase, and have the newer post break version save it !
(This of course would require some generating for some things ,like if trees go from single to multi blocks, which could be perhaps done with some generating code :) maybe write a script to delete old trees, after saving their locations, and generate the root/trunk in the same place, the games already present tree generating code would be optimal, and match better the way it generates things of course .

This is much harder than you made it sound. For one thing, the "ram data" is not compatible between DF versions either. The code itself is looking for different data structures in memory, so just copying in data from another version will certainly crash DF. In order for us to even come close to succeeding, we would have to transfer each data field individually - and for many of them, we don't even know what they are or what they do, even in DF v50! We haven't bothered researching many (maybe even the majority) of data types in DF because they simply aren't relevant to our common use-cases, but they are very relevant to DF. So getting anywhere close to stable would require a huge amount of research, both into old and new versions of DF, and we generally prefer to spend our time on more immediately useful things. Someone motivated could probably manage to make a bare-bones converter that converts some things (e.g. transferring map tiles and unit data through some intermediate format), but I don't think making a complete converter is feasible.

Have you already tried or thought of this kind of approach? It would certainly make a huge difference is Tarn or Zach were to help. Have you tried asking Tarn and Zach (or I heard they have a new third helper?... putnam??? ) for help with this? Am I even thinking right in assuming youve tried to make save updating at all, or is that just me? I think my brain energy is going down now so if Im getting less concise and cohearent, please forgive me. Im going to just pause now.

Anything is possible with the virtual, with a wide range of circumstances and ease/difficulty.

Unfortunately, I doubt we would be interested in pursuing this approach, especially given the relatively few requests we've gotten for it over the years. I think most people who want to play saves from old versions just play those old versions.

[EmotionallyBroken]

Originally posted by lethosor:

This is much harder than you made it sound. For one thing, .

Well I didnt say it wouldnt take any work or effort. I just said "maybe the easiest" :)
Your explanation of what would be a practical is very of implementing this general concept is very logical and has informed me of at least the general idea of what is already known, and what is not known by the two ,(or more?) people who are probably the most informed and aware of DFs mechanics and design along with computer software and at least basic(and perhaps much more) architectural interaction, Ive learned a few new terms (and more importantly actual standards of use and practice ) myself such as data serialization and memory data structures and buckets.

I am unsure, but I think the DFHack tools can alter all world terrain (or most of it), and maybe create units, even better if details about those units are fully alterable, though the basics only would still be a huge gain over not having anything at all from earlier version saves.

This makes me wonder though about the remaining things that are not implemented into the new play session and save state, and if missing data that is not yet known and created/coped would create a critical crash, or have smaller ..... amusements.

Originally posted by rome of oxtrot:

as to "updating an old save to load in a newer version", don't even bother: unless your name is Tarn Adams, you don't have anywhere near enough information to do this, and the fact that Tarn didn't put it into v50.01 leads me to think that he didn't either

I had simply assumed he didnt want to, and/or had introduced new mechanics that made that difficult like the tree mechanics.

Which again, makes me wonder, have you tried asking Tarn or Zach (or thier new co coder? putna---- ok I looked it up and here https://www.reddit.com/r/dwarffortress/comments/zst1ns/putnam_creator_of_the_dragon_ball_mod_among_other/ http://www.bay12forums.com/smf/index.php?topic=100799.0
word on the ... web is Tarn does all the coding and is not very open about his code for whatever reasons, however Putnam, a thorough mod maker himself may share an interest in helping to work on such a project or other collaborations with the DFhack team. There may of course be other interests and priorities , and thats a choice that is up to them , but here is hoping. :) If you have, when is the last time you tried talking to Tarn or anyone them about this?

EDIT:

Originally posted by lethosor:

Unfortunately, I doubt we would be interested in pursuing this approach, especially given the relatively few requests we've gotten for it over the years. I think most people who want to play saves from old versions just play those old versions.

Thats disappointing to hear, I imagine many others feel the same way I do, and would like to see the new mechanics and other upgrades in their play session, how much do you think its worth working on with your personal point of view? Whats your level of interest in this?
Whatever the answer to this, youve given me a lot of helpful, relevant information, with a logical, honest, practical and communicate way, and thats very much appreciate , particularly in contrast to the negative, uninformative, or sometimes even condescending responses I was expecting and usually see when asking about anything that takes effort :)

[EmotionallyBroken]

Originally posted by rome of oxtrot:

While I could respond to most of the above, I'd rather not do that in this forum; Steam discussions are just about the worst possible place to discuss in-depth technical matters. Feel free to join the DFHack development discord if you wish to discuss any of this in more depth.

While I very much appreciate your invitation, and your honest thoughts on the current medium, I dont have a discord account.