Palette data (*.raw) file structures

[dwtietz]

Just trying to get a bit of an understanding of all of the .raw palette data files and how their structure compares/relates to a standard palette.dat file such as the one used by Duke 3D. One thing for sure is that there does seem to be waaaayyyyyy more data here than in the old games.

Below are some of my guesses so far based mostly on file sizes, and I'm curious to know if anyone could shed any light, or provide clarification and corrections to my initial hunches. The files below all refer to those found in IM EAP v4.1.

There are bound to be some errors in my current understanding at this point, and I'm just wondering if anyone else has sorted some of these details out already.

File Name: 3dr_480.raw
Equivalent of the Display Palette (first 768 bytes) of palette.dat

File Name: ?
palette.dat has a two byte Shade Table Amount immediately following the Display Palette, but where is this in IM/IF? Is it hard coded into the executable instead or derived in some other way?

File Name: palookup_000.raw
Equivalent to 32 Shade Tables in palette.dat

File Names: palookup_001.raw ~ palookup_117.raw
Equivalent to a Shade Table in palette.dat (but 102 of them? Seems like an awful lot)

File Names: palookup_118.raw ~ palookup_124.raw
Equivalent to 33 Shade Tables in palette.dat (but 7 of them?)

* Of the above three entries, why would some seem to be a collection of shade tables in one file, while numerous others would be individual shade tables per file? All combined together, this would equal 365 shade tables which seems a bit excessive to me...

File Names: blendtable_000.raw ~ blendtable255.raw
Equivalent of Transparency Data Table found at the end of palette.dat (but 35 of them?)

File Names: palookup_125 ~ palookup_141
Equivalent to individual Palette Swap Tables in lookup.dat (But if so, where is the palette_swap_index that would typically preceed the table data?)

File Names: basepalette_000.raw ~ basepalette_007.raw
Equivalent to Alternate Display Palettes found in lookup.dat

[The Dukenator]

You should wait for the modding tools to be released.

[dwtietz]

Originally posted by The Dukenator:

You should wait for the modding tools to be released.

Why wait to develop an understanding of the file format data structures?

[mblackwell]

We have multiple pal swaps, shade tables, lookups, etc in Ion Fury. Yes, in fact hundreds of alternate palette settings for a surface.

[dwtietz]

Originally posted by mblackwell:

We have multiple pal swaps, shade tables, lookups, etc in Ion Fury. Yes, in fact hundreds of alternate palette settings for a surface.

That's some real next level crazy! I like it. :P

Lookin' forward for figuring out what's what in all this data dump. :)

Edit: I think that I'll be able to figure a lot of this out by inspecting the palette.def file. It looks like it probably contains some fairly straight forward information in here.

[mblackwell]

Yep, it's just good ol EDuke32.

[dwtietz]

Originally posted by mblackwell:

Yep, it's just good ol EDuke32.

Which is still actually fairly new territory for me. Back in the day, I had tried the first couple of releases of EDuke32, but I never got into it much more than that - Not because I didn't like it... Just because I was old school, hard core, original platform oriented at the time.

I certainly never got into inspecting and dissecting the structures within any of the file formats and how they had been vastly expanded upon. Clear similarities in the structure are still there, but the amount of data involved is substantially greater, and divided among numerous files.

[Fox]

These are common .raw image files, Photoshop can import/export them.

Example of base palette:

https://i.imgur.com/ao0EOp1.png

Example of lookup table:

https://i.imgur.com/BL7pjq7.png

Example of blend table:

https://i.imgur.com/nup83J2.png

[dwtietz]

Originally posted by Fox Martins:

These are common .raw image files, Photoshop can import/export them.

Cool. Do you know if these can be directly imported into GIMP as well? Less of a chance that Paint.Net could work with these files natively I suspect, so I'll probably need to perform some conversions for use with this application.

EDIT: Ignore the http link created by the name Paint<Dot>Net above as that isn't the website for this application. The actual website for Paint<Dot>Net is https://www.getpaint.net/

[Taamalus]

Originally posted by dwtietz:

Originally posted by Fox Martins:

These are common .raw image files, Photoshop can import/export them.

Cool. Do you know if these can be directly imported into GIMP as well? Less of a chance that Paint.Net could work with these files natively I suspect, so I'll probably need to perform some conversions for use with this application.

EDIT: Ignore the http link created by the name Paint<Dot>Net above as that isn't the website for this application. The actual website for Paint<Dot>Net is https://www.getpaint.net/

Yes, GIMP should be able to deal with .raw files. I stress should! .raw is comes in different formats. And I have not been able to test Ion Maiden's raw format.

https://howtogimp.com/raw-photos-in-gimp/
and the raw converter
https://sourceforge.net/projects/ufraw/
the peeps at Gimp suggested this instead
https://www.darktable.org/

another pro software is
https://www.paintshoppro.com/en/
No subscription, easy to learn. Affordable.

[dwtietz]

Originally posted by Taamalus:

Originally posted by dwtietz:

Cool. Do you know if these can be directly imported into GIMP as well? Less of a chance that Paint.Net could work with these files natively I suspect, so I'll probably need to perform some conversions for use with this application.

EDIT: Ignore the http link created by the name Paint<Dot>Net above as that isn't the website for this application. The actual website for Paint<Dot>Net is https://www.getpaint.net/

Yes, GIMP should be able to deal with .raw files. I stress should! .raw is comes in different formats. And I have not been able to test Ion Maiden's raw format.

https://howtogimp.com/raw-photos-in-gimp/
and the raw converter
https://sourceforge.net/projects/ufraw/
the peeps at Gimp suggested this instead
https://www.darktable.org/

another pro software is
https://www.paintshoppro.com/en/
No subscription, easy to learn. Affordable.

Thanks for the info Taamalus. I'll check that out. :)

I used to love using PaintShop Pro in the early days when it was free. I abandoned it when it became an application that needed to be purchased though. Took me a long time to find something to replace it with something else that is easy to use, versatile, yet free. I was very happy when Paint.Net eventually showed up on the scene.

I tend to gravitate toward GIMP if I need to do something more powerful or complex, but this is seldomly the case as I don't actually get into anything that I would consider being even close to "professional" level image manipulation, but there are some things that Paint.Net can't do that GIMP certainly can.

[Taamalus]

@ dwtietz - please, don't thank me yet.
The catch will be if Ion Fury requires the modder to exposrt actual .raw files. If Yes, you need Photoshop, since it seems the only software able to export Photoshop .raw files. :)

[dwtietz]

Originally posted by Taamalus:

@ dwtietz - please, don't thank me yet.
The catch will be if Ion Fury requires the modder to exposrt actual .raw files. If Yes, you need Photoshop, since it seems the only software able to export Photoshop .raw files. :)

No worries... The info still points me to a couple of avenues to try and poke around to see if it works or not. Even if it doesn't, it's still valuable info either way. ;)

EDIT: And... If need be, I could probably manually convert palettes in a Hex editor. This is the method that I'm most accustomed to viewing and manipulating palette data anyway. But if there are tools out there that might be able to do the job, all the better.

[Fox]

I'm not familiar with GIMP or Paint Net. But .raw format should be the easiest one to edit in Hex.

[mblackwell]

Yes they import into GIMP just fine.

[dwtietz]

Originally posted by mblackwell:

Yes they import into GIMP just fine.

Yay!