Reduce filesize down to 1/5 = re/zip EVERYTHING into .ap

[Adam Beckett]

Usually, a good compression rate is 2:1 for files, in general. Steam itself is trying to deliver all files at the highest compression rate, saving bandwidth for both sides, but does not succeed beyond 2:1.

Enter "RailSimulator/TrainSimulator"...

When zipping routes, especially, the compression rate is more like 3:1, 4:1, even 5:1! An 'unpacked' route can take up let's say 1.5 GB and zipped (renamed to .ap), can be just 400 MB 'big'!

Two reasons:

Non-compressed Train Simulator is eating up your hard drive - as in 'clusters' (=equal-sized chunks, which are created after formatting) and nodes (index data structures), which are created for every single(!) file explicitly written.

Basically, the more files you write on a hard drive, the more clusters you are wasting because you are creating more 'overhead' (Windows file system has to search and find each single file via the NTFS tables).

Second - more dramatic - reason:

... file redundancy in RW/TS!!!

If you ever installed a reskin, you are familiar with the concept of reskinned 'locos' being a copy of the original 'default' folder with custom texture files and the infamous "Copy ***.GeoPcDx" mesh file.

This is the very definition of 'redundancy'. And compression gets rid of all those manifold, multiple 'same' files.

'Video games' use containers and compression for a long time. Most modern video games are efficiently compressed where possible and come in containers named .pak or similar names.

Writing ONE BIG file, instead endless tiny ones allows the Operating System to read once and open the data in system memory for the busy business, instead reading (and writing) again and again and again, exhausting the needles of your physical hard drives.

DTG seemed to have realized the benefit of compression, since the started shipping main folders as zipped files, renamed .ap. Thankfully, this is no obfuscated format, but easy to use - since it is using zip - and the contents are still easy to read and manipulate.

Now, what I propose YOU do, is ZIP ALL THOSE 3RD PARTY ROUTES! Safe space (SSD) and prolong the life cycle of your HDD, if you still use those.

1. Go inside each Route ID main folder (for example "2a941a3f-5fdc-4082-ab8b-a083664aeb9d")
2. Select all files and folders
3. right-click "send to zip" (or zip-tool of your choice: 7zip, WinRAR, WinZIP ...)
5. Rename the created 2a941a3f-5fdc-4082-ab8b-a083664aeb9d.zip file into MainContent.ap
6. It works!

That's the thing: you can zip and rename to "MainContent.ap" EVERY - non-DTG route(!). "MainContent.ap" is recognized by RailWorks.exe and runs perfectly fine, no matter which Provider or user-mod.

Equally, re-zipping Asset folders, which you may have unzipped to add 200 reskins is equally beneficial. This is as much a reminder to myself, as it is to maybe you. Even those newly created reskins can be zipped into the original AssetPack.ap container, reducing the redundancy incrementally and incredibly, since compression-algorithms do not write the same file twice (... or 50 times - for 50 loco reskins!), but only LINK to the position of the one necessary file 50x instead storing 50x equal files (I cannot repeat this often enough).

And that's just the 'easy' stuff compression algorithms do. They only start here and move to pattern recognition, etc.

Asset folders which are not from DTG and do not come in a .ap format, are sadly not zippable this way, since the ScenarioProperties.xml file links directly to the assets inside and would not recognize a newly created parent-folder?

I put a question mark there, since I am trying to figure out if there is a way. Assets folders in particular are filled with redundant files, due to the 'copy+paste' antiquated nature of duplicating locos for every reskin, brand-patch or headcode (at least, SOME Providers figured out 'scripting' and are doing it more efficient. But that is a drop in the ocean, if you are hitting 1 Terabyte of Assets).

At some point (switching to SSD), there is a cost-benefit analysis answer which says: "Why waste your time? Just buy more space?". Valid point. But that is never a programmer's answer.

PS: Any ideas on how to solve "the non-DTG Assets problem" are welcome! One way would be, of course, to zip those Asset folders anyway and then edit every single(!) ScenarioProperties.xml file ... by hand. Way too time consuming, nor an 'elegant' way, for such an old game with thousands of scenarios.

[Phase3]

Good points about the bloatware aspects of TS files.
I was informed by RSC/ DTG that .ap files are zero compressed files so a 1 GB .ap file will extract to 1 GB and that extracted file is "read" first by TS. Is that not rue now? I just extracted the Fife Circle route the .ap file was 239MB and the extracted files totalled 239MB.
The main reason DTG use .ap files is for checking the integrity of the game files whilst TS is loading - it is easy to verify a single .ap file rather than hundreds of files as was done pre .ap files before 2014.
On loading (procmon) TS runs around 14 million lines of code including checking all of the .ap files installed.
Now if you use a zip compression ratio of more than 1:1 do the files still work OK efficiently?
Surely if you use a higher compression ratio it is going to take longer to extract them for use in TS?
I am fortunate to use SSD's for my TS install and so I don't mind there being lots of files as I have a lot of free space.
TS ONLY loads one route and one scenario PLUS associated assets at anyone one time - so I am not sure what you would save by compressing files except disk space.
Plus if you have sufficient RAM installed - once the TS working set is loaded via the OS and VAS - it makes little difference to the running of TS whether you use a zip, or .ap file or not.

Thanks for a thought provoking well written article!

[Adam Beckett]

Very interesting. Thanks for your reply.

I checked the .ap files and indeed - as you said - DTG's .ap files are 1:1.

I did not know that.

I used so far 7zip 64-bit for the non-DTG files with a 'default' (= variable) compression rate.

For example:

RSSLO Karawankenbahn
unzipped 530MB
zipped 288MB

JustTrains Chiltern Main Line
unzipped 704MB
zipped 204MB

[BHBP] Berks & Hants to Bristol & Penzance v1
unzipped 2.1GB
zipped 508MB

etc.

So far, as I am still in the process of testing and zipping routes, I cannot see any flaws or 'missing' parts. All works perfectly fine. But, I do need to play more, to make sure. It seems the compressed .ap file is 'accepted' and being uncompressed during loading into memory, without complaint.

I did not measure the time it takes to load any scenario with everything compressed (not just stored) into MainContent.ap. I really doubt it makes any difference in performance or length-of-time-to-load on todays (last 10 years) hardware?

You are also right about the SSD aspect!

There really seems little point to touch any of the files and folders when you are using any kind of SSD (SATA, PCIe, ...).

What made me do this in the first place, is that I exceeded my 1TB TS20xx space limit

I was thinking about how to reduce the size of the files, knowing about all this 'wasted' space - especially, in the Assets/RailVehicles folders - having endless copies of the same old files around (especially, with the vast amount of reskins, we all gathered).

There are ways to do this (either self-written scripts linking to the files/folders or symbolic link/junction to the default files/folder, instead copy+pasting all files to the reskin root loco folders), but all take a lot of manual labor. Nothing can be 'automated', afaik. And any rare new 'update' could break this house of cards instantly.

So - again - the "is it worth the time?" question tells me 'no'. Especially, if nobody else can benefit from this, without having to do all of it themselves. It is hard to put the Genie back into the bottle (= make structural changes to an 'outdated' game as a user).

[Phase3]

Adam
Keep these discussions going - I really enjoy reading them!

[Adam Beckett]

[UPDATE]

Achievement: Further was able to reduce the installed game to 60% of it's original size.

SSD capacity: 1.000.188.407.808 bytes - 931 GB

Before compression:

Used Space 985.409.957.888 bytes = 917 GB
Free Space  14.778.449.920 bytes  = 13.7 GB

After successful compression

Used Space 582.962.974.720 bytes 542 GB
Free Space 417.225.433.088 bytes 388 GB

Requirements:
TS20xx installed on an SSD (or equal non-HDD)
Windows 10
Modern CPU
> 8GB System RAM

Used Compression Method:
Windows 10 NTFS Disk Compression

Explanation: Since the beginning of PC gaming on Windows the ONE 'tabu' was always to NEVER use disk compression on partitions you want to install games. Even to this day many games on Steam are writing in their hardware requirements that their game needs 'uncompressed' disk space, to run.

What has changed?

CPU's are significantly faster, making additional loading times insignificant.
Compression algorithms got better, speeding up read/write actions.
Running out of memory while doing decompression (reading files) is not an issue.

There are articles and discussions on the web about performance and usability of this method, you can googlebing, if you are interested in the topic in general.

A game like RailSimulator/TS20xx is a champion of wasting space. The endlessly redundant files are written again and again in every loco-livery sub-folder, or routes folders. The ONE saving grace - as stated at the first post in this thread - is zipping folders into .ap containers. But this is not done everywhere, by everyone.

Why do you want to do this?

I myself had an uncompressed TS20xx folder which exceeded 1 Terabyte.
After zipping all content I was sure can be zipped into .ap containers AND after compressing the complete TS20xx SSD drive, I regained enough space to be able to either fill it with more content.

The performance aka additional loading time length is not measurable on my system (2nd Gen Ryzen - Ryzen 7 2700X).

Another overall performance gain would be to use a NVMe SSD over the SATA version, to speed up loading times further. 1 TB NVMe SSDs are now not more expensive than SATA SSDs. (Check Amazon or Western Digital in particular).

[Felix.AVMP]

There is, of course, another caveat (fairly obvious, though) - if you do this on DTG provided content, you should under no circumstance run game files verification...


Regarding file compression - there is an interesting paradox at work. While compression takes CPU cycles, file compression also means, that hard drive (no matter what technology it uses) has to transfer lower amounts of data. As a result, occasional speed ups were detected in the past anyway, even on single core CPUs, however, it always required testing as it always was on case by case basis and it was almost impossible to predict the results on different hardware.

Btw. Steam requires uncompressed disc space, because it is practically impossible to predict exact compression ratio, that it the sole reason, why Steam is using this condition.

But again - this is an interesting topic and thank you for that!