.pak files vs RAM. Video Soon...

[SUDTIN]

The problem described in this thread is not isolated to the Open Beta and is a precursor of problems to come with the September 28th launch so I feel it is worth our time to address the biggest flaw of the New World Open Beta. The .pak files.

New World Open Beta is divided into more than fourty .pak files that are slightly more than one gigabyte in size. The main content .pak files that hold multiple quality levels of textures and objects are scrambled and uncategorized so badly that when the game calls for a single texture or object; the CPU must reload an entire .pak file into RAM and overwrite a previous .pak. It's a problem because New World Open Beta will fill the RAM with every .pak file it can from the very first moment it's launched and endlessly overwrite previous .pak files it has already stored into RAM continuously and endlessly even after the RAM is full with zero free space left.

So as an example while playing New World Open Beta; the CPU persistantly reloads these .pak files over and over and over for single textures, sounds, objects or code that it seemingly forgets are already stored somewhere within the RAM. This can be triggered just by changing a shirt or walking into a town. Instantly a "streaming data" call is made from the Game to the RAM for a single object and or texture but if it the object and or texture isn't there in the RAM then it the RAM forfeits a different .pak it's not currently using and overwrites that different .pak for litterally just one object and or texture.

Every .pak stored in less than 64GB of RAM is subject to "streaming data" calls and can be overwritten based on the need of one single asset.

So now we see the huge problem with Lumberyard and why Amazon is moving on to a different engine. Instead of the game reading the .pak file and storing the one asset it needs into RAM. The game brings entire .pak files into the RAM as it endlessly overwrites itself based on "streaming data" calls for single asset needs. The engine is essentially flooding the RAM with data it cannot read and will never use and will endlessly overwrite the RAM until the end of time...

Look for yourself.
Start > All Apps > Windows Administrative Tools > Resource Monitor > Memory or Disk

https://youtu.be/AnS7XsCFYSU

Unless there is a way!

Everyone experienced problems during the Open Beta. Many theories floating around as to why and what caused them. Many even arguing that there are no problems at all... After understanding the type of problem that plauges New World Open Beta. There is one group of players that saw the least amount of problems. Players with more than 64GB of RAM. The game put every single .pak file into RAM and there was room left over. Maybe at that point the game runs purely from the RAM and never makes another "streaming data" call that causes the Game to endlessly overwrite the RAM searching for single textures...

Key problems that could be different. Solution?!?

New World stores every .pak file into RAM the moment the game is launched. Averted with more RAM. Minimum 64GB.

Solution is to categorize .pak files into Low Medium High Very High and only load needed .pak files based on video quality. Reduce the .pak load on RAM.

New .pak categories look like this.

Soundtrack.pak
AmbientUICombatSound.pak
TexturesLow.pak
TexuresMedium.pak
TexturesHigh.pak
TexturesVeryHigh.pak
TerrainLow.pak
TerrainMedium.pak
TerrainHigh.pak
TerrainVeryHigh.pak
FoliageLow.pak
FoliageMedium.pak
FoliageHigh.pak
FoliageVeryHigh.pak
DetailLow.pak
DetailMedium.pak
DetailHigh.pak
DetailVeryHigh.pak
UI.pak

About all the game needs to be fully loaded on High without the Soundtrack would look like this.

AmbientUICombatSound.pak
TexturesHigh.pak
TerrainHigh.pak
FoliageHigh.pak
DetailHigh.pak
UI.pak

Maybe 8 to 16 GB.

Hope for the best but expect the worst.

Seperate all Low from Medium from High from Very High because that alone will DRASTICALLY improve all issues.

"CryPak File Archives

CryPak is a module which enables to store game content files in a compressed or uncompressed archive.

Features

Compatible with the standard zip format (implementation is based on the zlib library).

Supports storing files in an archive or outside in the standard file system.

Data can be read in a synchronous (stalls until the data is loaded) and asynchronous (streaming) way through IStreamCallback (max 4GB offset, 4GB files).

Files can be stored in compressed or uncompressed form.

Uncompressed files can be read partially if required.

File name comparison is not case sensitive (internally everything is converted to lower case).

Supports loading of zip/pak files of up to 4GB in size (proven to work with more than 2.1GB)."

https://youtu.be/Ciu7xvqAk1k

Devs should read this and properly pak their assets for launch.
https://github.com/o3de/o3de.org/blob/main/content/docs/user-guide/packaging/asset-bundler/command-line-reference.md

[nelvus]

Truly a marvel of engineering.. if it's true.

[Subsonic]

Did you check this along side pagefile usage and size ? Maybe just increasing the pagefile space will make a difference on streaming read calls.

There's plenty of ways they could reduce streamed loading requirements.

[SUDTIN]

@Evol Yes I tested for GPU and Pagefile usage. New World Open Beta didn't use the Pagefile whatsoever. New World is also 100% incompatable with Superfetch and will fill the RAM the same way when the Sysmain Service both on or off.

[Liliana M]

On my system NewWorld.exe was throttling my hard drive I/O by reading pak files over and over at full read speed.

[starkmaddness]

Interesting. Game performed flawlessly on my machine, but I have 64GB and a fast NVMe drive. So makes sense.

Though I disagree on your suggested fix. Usually assets are grouped by biome/zone/region. You don't want to load every single tree texture in the game into memory when your zone uses only a small fraction of the trees. The same with every other asset.

Even separating them by high/low, etc could be an issue if they use LOD for textures. Just because your game is set to HQ textures doesn't mean it is using those 4k texture for objects 500 meters away.

[SUDTIN]

@starkmaddness Yes LODs would need their own asset .pak and must be loaded for every quality preset.

Maybe it would need some reverse engineering... Perhaps if it works like you are suggesting. The base game Low textures objects etc are the furthest LODs. Then the option to disreguard and not even load Medium High and Very High .pak files would be an alternative performance fix.

The loading and storage of .pak files into the RAM is a work around for Lumberyard because it can't read and store one single asset from an Index of .pak files. It must copy the entire .pak file into RAM in order to retrieve even just one asset. Lumberyard could technically read single assets if the crypak file system isn't used. Otherwise it is incapable.

[Subsonic]

Originally posted by starkmaddness:

Interesting. Game performed flawlessly on my machine, but I have 64GB and a fast NVMe drive. So makes sense.

Though I disagree on your suggested fix. Usually assets are grouped by biome/zone/region. You don't want to load every single tree texture in the game into memory when your zone uses only a small fraction of the trees. The same with every other asset.

Even separating them by high/low, etc could be an issue if they use LOD for textures. Just because your game is set to HQ textures doesn't mean it is using those 4k texture for objects 500 meters away.

Well yeah they could use a number of techniques, they might not even be using lower resolution textures and generating mipmaps on the fly from High Res ones, would have to do some digging and some testing to see what they are actually doing.

[Svanlaug]

Also flawless performance on my machine, 32GB DDR4 3200MHz, SN750 NVME M.2 SSD.

[Logisk]

After reading this i wonder how i could run New World without performance issues using 6 years old hardware and having the game installed on my mechanical disk :P

It most be executed in a really good manner for it to work that good on a mechanical disk :P

And it never used more then 8GB RAM i think, i have 64GB RAM

[starkmaddness]

Also, New World is not using O3DE, it is using Lumberyard.

They locked in to a version of Lumberyard before it transitioned to O3DE.

(Link to an Interview with Director of the game.)
https://venturebeat.com/2021/09/13/new-world-interview-how-amazon-games-brought-an-mmo-close-to-the-finish-line/



They are not the same thing. O3DE is essentially a different engine not simply a version change.

"And no, O3DE is very different than the artist formerly known as Lumberyard."

(Link to Amazon talking about O3DE)
https://aws.amazon.com/blogs/gametech/open-3d-engine/

[SUDTIN]

@starkmaddness

Yeah Amazon removed the multiplayer parts of Lumberyard before they put it up onto GitHub as O3DE.

What is your point though. It's the same .pak system obviously. Might as well just call it CryEngine.

@Logisk

It makes sense that a Windows 7 PC wouldn't have a problem considering everything Windows 11 is backwashing into Winodow 10 currently... Must be something wrong with my computer if I am the only one.

[Svanlaug]

Interesting article covering some background. https://venturebeat.com/2021/09/13/new-world-interview-how-amazon-games-brought-an-mmo-close-to-the-finish-line/

[TWiZTED]

Hope this reaches Amazon and devs. Great thread.

[Pwesty]

I can name a few F2P /P2W games that use pak files the same way and dont have this
"issue" i have 16gb of ram never had any issues with overly used ram or from what sounds like a constant looping memory leak that gets to a certain point and refreshes itself. ive played both beta's and again not many issues.

while you are correct about pak files "im well hated and known on CERTAIN forums"
i think the issue isn't really the file itself just how they are handling it.

initial load up requires decryption of said pak files, files get checked, then dumped into memory "maybe a crap anti-cheat system is doing this" then if all goes well you get into game.
Some anti-cheat systems will constantly recheck files to make sure nothing is being tampered with.
this is my speculation on this of why its doing what it's doing.
you put the check on a function that's already loaded "send to server" and load it with a check function every X mins, you're going to have issues.

i can't say my background in game development "personal and TOS reasons" but i've seen this happen exactly like you described it. but again i dont have any issues of sort and i have a 6th gen intel system w/ 16gb of ram.

[Odd World]

It would be nice if New World used more of my VRAM.

Totally unrelated to all the nonsense you just posted, I'm just adding this as a side note.

https://github.com/aws/lumberyard/tree/master/dev/