Strings

[Frankelstner]

Use this hashlist: https://bitbucket.org/Frankelstner/pd2-hashlist/overview

Thanks to Zdann for resolving many tough material_configs.

Background:

As you may not realize, Overkill has removed the file with strings (filenames among them) with U100. The actual archives only specify a hash, and the hashing function is known.

To extract files, take all strings/filenames and calculate their hashes, then make a table out of this info. When the archives then mention a hash, we look it up in the table and get the actual filename. Obviously, this only works when we have the filename in the first place.

The removal of the strings file makes no sense whatsoever given the mod_overrides folder: http://steamcommunity.com/app/218620/discussions/15/619569341120461778 I believe you need to know the filename to override. In any event, filenames make navigation a bit easier.

Only few people noticed, but in my opinion the removal of the strings is even worse than the initial micro transactions.

Anyway, I have determined most strings.


Methods:

The basic idea is to get a list of all hashes that cannot be resolved. Then find a suitable pool of strings (about 10^8) and calculate their hashes. If a calculated hash is found in the unresolved list, success; we add the corresponding string to our hashlist. If a hash is not found in that list, just ignore it and try the next. So the goal is to find a good pool of strings.

1. I grabbed all strings from all bundles and decrypted Lua files with regex. This lowered the unresolved files from 11000 to 6000.

2. I took the existing_banks.banksinfo file and read the two string sections. I don't understand the exact format, so I just took one string from each section and combined them in all possible ways. I also went through the individual bnk files and grabbed whatever looked like an integer (which becomes part of the filename); this alone resolved only about 50% of the stream files. I'm not sure if that second approach was necessary at all. Anyway, all streams resolved.

3. The Lua concatenates strings for various icons. So I just go through the corresponding tweakdatas and combine DLC + object name to get a string.

4. The Lua gives the path to each mission folder. I try the common filenames like "world", "continents", ... and then run my parser on these files, which gives me some data structures to traverse. From those I determine the subfolder names. Then I parse the continent files in the subfolders to get all statics that have light (which corresponds to the cube_lights subfolder), as well as all instance names. For each instance I then do step 4 again (except for the first sentence).

I also tried adding _cc, _thq and _thq_cc at the end of each file because those material configs are really hard to come by.


If you have any info about the remaining files, let me know. The exact filename or even the calculation in the Lua would be great.

[Frankelstner]

Scripts used (Python 2.7):

Hashing function wrapper: http://pastebin.com/PAGMxLK6 Requires the hashing function dll: https://www.dropbox.com/s/c1mwqvn8mow3de7/payday2hash64.dll?dl=0
Archive tools: http://pastebin.com/sbtDtGWN When run directly, extract all files in D:/mydump.
Mission parser: http://pastebin.com/vZnAxYq0

Main: http://pastebin.com/CGFeQzir

[Zdann]

Not quite done extracting the bundles yet, but I've noticed some of the missing files are the contour material configs, which are typically only for NPC weapons and character models. They have _contour appended at the end of the name, like ene_zeal_swat_heavy_contour.material_config.

[Frankelstner]

Thanks, already found some more. The full regex is going to take a few more minutes though.

*removed, use link in first post*

[Frankelstner]

I think only vehicles are missing from the material_configs now. Textures are still a wild mix of everything. And there are just 4 (probably empty) Lua files and 4 other files missing.

[Zdann]

Few more material configs I found by looking at a sequence manager:

units/pd2_dlc_chill/characters/shared_character/mtr_chains
units/pd2_dlc_chill/characters/shared_character/mtr_hoxton
units/pd2_dlc_chill/characters/shared_character/mtr_dragan
units/pd2_dlc_chill/characters/shared_character/mtr_jacket
units/pd2_dlc_chill/characters/shared_character/mtr_old_hoxton
units/pd2_dlc_chill/characters/shared_character/mtr_wolf
units/pd2_dlc_chill/characters/shared_character/mtr_john_wick
units/pd2_dlc_chill/characters/shared_character/mtr_sokol
units/pd2_dlc_chill/characters/shared_character/mtr_jiro
units/pd2_dlc_chill/characters/shared_character/mtr_bodhi
units/pd2_dlc_chill/characters/shared_character/mtr_jimmy

[Frankelstner]

Huh? Good find! Gotta investigate why my script did not pick that up. (Tomorrow.)

[Zdann]

And the van skins:

units/pd2_dlc_chill/vehicles/str_vehicle_van/str_vehicle_van_brown_chill
units/pd2_dlc_chill/vehicles/str_vehicle_van/str_vehicle_van_green_chill
units/pd2_dlc_chill/vehicles/str_vehicle_van/str_vehicle_van_grey_chill
units/pd2_dlc_chill/vehicles/str_vehicle_van/str_vehicle_van_red_chill
units/pd2_dlc_chill/vehicles/str_vehicle_van/str_vehicle_van_white_chill
units/pd2_dlc_chill/vehicles/str_vehicle_van/str_vehicle_van_yellow_chill
units/pd2_dlc_chill/vehicles/str_vehicle_van/str_vehicle_van_icecream_chill

units/pd2_dlc_chill/vehicles/str_vehicle_van/no_skinning_materials/str_vehicle_van_brown_chill_no_skinning
units/pd2_dlc_chill/vehicles/str_vehicle_van/no_skinning_materials/str_vehicle_van_green_chill_no_skinning
units/pd2_dlc_chill/vehicles/str_vehicle_van/no_skinning_materials/str_vehicle_van_grey_chill_no_skinning
units/pd2_dlc_chill/vehicles/str_vehicle_van/no_skinning_materials/str_vehicle_van_red_chill_no_skinning
units/pd2_dlc_chill/vehicles/str_vehicle_van/no_skinning_materials/str_vehicle_van_white_chill_no_skinning
units/pd2_dlc_chill/vehicles/str_vehicle_van/no_skinning_materials/str_vehicle_van_yellow_chill_no_skinning
units/pd2_dlc_chill/vehicles/str_vehicle_van/no_skinning_materials/str_vehicle_van_icecream_chill_no_skinning

[Ellie]

This is super helpful, thanks c:

[Frankelstner]

Guess I should not have allowed ' in the regex, because then I got 'units/...' instead of just units/... and then it did not hash the right thing. Anyway, hashlist updated. 20 material_configs left. 103 textures left. 4 tiny Lua files and then 4 other files. Looking good.

[Frankelstner]

I got a new string from the U104 Lua: guis/textures/menu_title_screen_sale

This suggests that the bundles contain assets that are not referenced anymore or have never been referenced. So there is a limit on the things we can resolve. On the bright side, the unused assets are all placed next to each other in the main folder (with the hashed name, sadly).

Of course, I have probably missed some textures. Any confirmed ingame sighting of any missing texture is greatly appreciated.

Is this one actually used? http://i.imgur.com/ABnMTLy.png

[Frankelstner]

Some more strings. 100 files missing. Some of them are just duplicates of correctly resolved files; the burn-offshore trophy exists twice. Can the unresolved mask materials be accessed already?

[Pola]

引用自 Frankelstner：

So there is a limit on the things we can resolve.

If it's just git related couldn't we use one of the decompiled lua git repo's and run the script through each tagged commit? Too bad no one keeps the assets version controlled I guess?

You seem to have the majority of it sorted from the looks of it. Not sure if there is much point in finishing/releasing my GPU cracker :P It's good for guessing 9ish characters, after that length it'll start to become impractical. It'd be fine with guessing audio files that have 32-bit int file names, especially since the folder names are identified by the soundbank files. Beyond that you need to apply knowledge of folder hierarchy/patterns. If I go back to working on it, using your handy script to feed the paths into my pattern analyzer would be good. Dictionary attacks with known words from all the paths could also help I guess.

[Frankelstner]

There is no guarantee that some assets have ever been referenced. I don't expect the older Lua to contain any more paths.

I get 10 duplicates; unresolved files with the same payload as a correctly resolved file. This includes some safehouse GUI textures and the GUI texture for the mask Blister Head. So apparently they just shipped the same asset twice, but one is in an unreachable spot.

What's left are some material_configs that are not referenced by any sequence_manager and thus probably unused as well. Those configs themselves make fairly clear where they belong.

E.g. f3d1be12403a1769.material_config (this should be big endian notation, contrary to the common bundle modder) says <diffuse_texture file="units/pd2_dlc_chill/props/chl_prop_godrays/chl_godrays_01_df"/>.

And we already have units/pd2_dlc_chill/props/chl_prop_godrays/chl_prop_godrays.material_config which has the same payload.

Actually after some attempts I have identified that file as "units/pd2_dlc_chill/props/chl_prop_godrays/chl_prop_godrays_small/chl_prop_godrays_medium". Thing is, this _medium suffix is already found, but only in the folder one level above. I guess the remaining files really just need some manual intuition + some fast dict attacks.

[Pola]

If you want fast dict attacks use the GPU. Very easy to do with Hashcat, I've got the hashing algorithm written in OpenCL using 64-bit vectors.

[Frankelstner]

Ah, I meant very simplistic attacks that need add just 1 word. But to be honest, I believe these remaining material_configs are truly not used at all. The textures are of greater interest.

All dialog files resolved: gamedata/dialogs/safehouse_raid
gamedata/dialogs/b_flat_stage1