[SOLVED] StudioMDL messing up weight paint

[Marco Skoll]

I'm having a really irritating issue with one of the meshes for a model (well, possibly more, but this is the one I'm noticing it on). The weight paint is changing between exporting from Blender and when it gets into SFM (this picture shows a re-decompiled version, but the issue is the same in SFM):
http://steamcommunity.com/sharedfiles/filedetails/?id=1149927802
The model weighs in at 4,765 verts and 18,828 tris, with one flex of 320 verts to fix clipping with certain clothes. The weight paint as exported from Blender is already culled to 3 weight links and normalised.

I've tried eliminating potential issues in the QC one by one (disabling procedural helpers, making sure this mesh is used as the reference animation, etc), but the issue persists even if the QC looks like this, with nothing in it other than this one mesh and its materials:

$ModelName "DMF\Vexia\Vexia_TestCompile.mdl"
$Body "Human" "vexia_bodyhuman.dmx"
$MostlyOpaque
$CDMaterials "models\DMF\Vexia\"
$Sequence "idle" "vexia_bodyhuman.dmx"

The weight paint works pretty well in Blender (I've been redoing the whole thing basically from scratch in order to get the model to work its best with Source's weight paint limits), it's just FUBAR after I try compiling it.

So, what exactly am I doing wrong here? I'm not expecting the model to work absolutely 100% the same as in Blender, but this is more of a discrepancy than I'm prepared to accept.

EDIT: Solved. StudioMDL by default culls bone weights of less than 0.05, which was throwing off the weight paint.
episoder came up with an edit to StudioMDL that could adjust this cut off point.

[Pte Jack]

Marco, Do you have any shapekeys that influence the messed up mesh? It could be that they have to be fixed a bit if you do...

[Marco Skoll]

There is one shape key on the mesh, but it affects the fit around the chest area, and doesn't affect the shoulders.
The problem persists even if the mesh is exported and compiled with no shape keys, either as a DMX or SMD.

I guess I could try exporting to something like FBX and then reimporting/exporting and seeing what happens, in case it's something weird that Blender is exporting to the mesh, but that seems like grasping at straws.

EDIT: No, exporting to FBX and then reexporting as SMD changes nothing. If there's any undesired data on the mesh, that isn't wiping it.

[Marco Skoll]

Further experimentation:

- The issue persists even if the model is exported to OBJ, loaded in a new session with an FBX, has the weights transferred across from the FBX, and compiled. So that basically entirely rules out that the error is specific to the original blend file I'm exporting from, it's something that persists to some degree across the OBJ, FBX and SMD/DMX formats.

- If I decompile with Crowbar and feed the decompiled mesh back in, the weight paint distorts further (although different vertices), so it's not like it's StudioMDL rounding off to a certain precision.

So there's just something about this mesh that StudioMDL is bloody mindedly objecting to. I may have to just stubbornly readjust the mesh layout in those areas and see that changes anything.

[Marco Skoll]

... and now I've tried:

- Converting the mesh across the shoulder to quads: no difference.
The vertex weights are being messed up exactly the same in both cases.

- Different bone parenting: no difference.
Originally the bones were chained Clavicle (paint for collarbone) -> UpperArm (unpainted) -> UpperTwist2 (paint for upper arm, a helper bone to improve shoulder positioning, particularly when using IK and you can't control the main bone rotations). I tried Clavicle -> UpperTwist2 instead.

- A different version of StudioMDL: No difference.
Exact same results from the GMod StudioMDL as the SFM StudioMDL.

[episoder]

weight link culling? the export option? have you tried to 'limit total' to 3 in the weight paint tools. then normalize. i have no idea but culling might just remove the lowest of more then 3 leaving a numeric empty while the weights would not add up to 1 then. the compiler could renormalize that. you could do a test with just the shoulder mesh temporarly seperated, and compare the output weights if you export to smd.

edit: ^ that is nonsense. both smd and dmx (dmx2tex checked) output normalized. i'm not sure how the compiler deals with too much weights in dmx tho. smd throws errors.

[Marco Skoll]

Originally posted by episoder:

have you tried to 'limit total' to 3 in the weight paint tools. then normalize

Yes. That's what I meant when I said I'd culled the links.

you could do a test with just the shoulder mesh temporarly seperated, compare the output weights if you export to smd.

I've not tested with just cutting that part of the mesh out yet, but compiling just that mesh makes no difference.

As far as SMDs, they're doing exactly the same thing.

EDIT: What I'll probably do tomorrow is export a version of the mesh from before I started messing around with it trying to get it to work, and upload it for others to experiment with.
(I'll also need to mirror the weight paint, as I'd only actually finished the left shoulder and was trying to test it in SFM - I still need to copy it over to the right).

[Pte Jack]

When you export obj then bring it back into Blender, the mesh is not affected? Only when exporting DMX/SMDs? Have you tried back stepping to an earilier version of blender and try the export. There may be inconsistencies between the new 2.79 and source tools. (Maybe??) Maybe something Artfunkel has to look at??

[Zappy]

Originally posted by Pte Jack:

When you export obj then bring it back into Blender, the mesh is not affected? Only when exporting DMX/SMDs? -

No, only when compiling a model using StudioMDL (and thus also when de-compiling it and getting SMD(s) from the de-compiled model, but not what's exported from Blender at first).

Marco Skoll has said that exporting an SMD and re-importing it gives the same result as before exporting it, but compiling it into a model starts giving these strange issues, which persist when de-compiling it.

[Smug Bastard]

Most likely issues you experience are due to compiler culling anything below 0.05 weight limit

[Marco Skoll]

Originally posted by Pte Jack:

When you export obj then bring it back into Blender, the mesh is not affected? Only when exporting DMX/SMDs? Have you tried back stepping to an earilier version of blender and try the export. There may be inconsistencies between the new 2.79 and source tools. (Maybe??) Maybe something Artfunkel has to look at??

As Zappy says, exporting DMXs or SMDs is itself fine. The picture at the start used an exported DMX loaded into a new session.

The issue only exists after compiling. (Or so it would seem. I suppose to an extent I'm trusting what Blender tells me the weight paint is, but Blender seems to agree with SFM about what's happened after the compile).

That said, trying an old Blender isn't a completely wild idea.

Originally posted by Smug Bastard:

Most likely issues you experience are due to compiler culling anything below 0.05 weight limit

If it were a case of StudioMDL "rounding off" to a certain precision or culling weak links, I would expect a decompiled SMD to be "valid" by those criteria and thus recompile the same. It doesn't - the weight paint distorts again.

In any case, comparing the paints, it's often not the weakest links being affected.

[episoder]

not sure if i missed that. you decompiled? it's possible the error is from the decompiler?

in that case you should test the model in sfm. and done. you shouldn't bother thinking about them other's ripper. it's not an important result to test thru decompilation. that a lil bs.

[Zappy]

Originally posted by episoder:

not sure if i missed that. you decompiled? it's possible the error is from the decompiler? -

Not if the error also occurs within Source FilmMaker (which, according to the opening post, it does).

[Pte Jack]

What I mean is, there is no way to check the model between its Blender/SMD-DMX state unless you actually import the SMD/DMX immediately after export. If the mesh is pooched on import, then it is a blender/Source Tools inconsistency, not a SudioMDL problem. But try Blender 2.76 or 2.77 to see if the problem persists. That too will give you ammo to use if it is a Source Tools problem to give for Artfunkel to review or should confirm that the problem lies in the model and StudioMDL.

[Zappy]

Originally posted by Pte Jack:

What I mean is, there is no way to check the model between its Blender/SMD-DMX state unless you actually import the SMD/DMX immediately after export. If the mesh is pooched on import, then it is a blender/Source Tools inconsistency, not a SudioMDL problem. -

Marco Skoll has stated that exporting a DMX/SMD and then re-importing it results in it looking the same as it did in Blender before, with the issue first happening on the compiled model and also files gained from de-compiling it.

[Marco Skoll]

Originally posted by episoder:

not sure if i missed that. you decompiled? it's possible the error is from the decompiler?

No. Or at least "not entirely".

The error is definitely in SFM to some extent, because I didn't even start decompiling to take a look until after the model had repeatedly failed to deform satisfactorily in SFM (and after I'd started suspecting that I couldn't have messed up the weight painting that badly five times in a row).

It's theoretically the case that in the picture I posted at the start that decompiling has exaggerated the issue by the time I'm re-comparing meshes in Blender, but it cannot be what is causing the issue in SFM.

Originally posted by Pte Jack:

What I mean is, there is no way to check the model between its Blender/SMD-DMX state unless you actually import the SMD/DMX immediately after export.

I have. That is exactly what you see in the first picture.

The pink mesh has been reimported into a completely new Blender session after having been exported as DMX from the original file.

The blue mesh is a SMD that has been compiled and decompiled from that same DMX.

Theoretically, Blender might be messing up the weight paint in exactly opposite ways when I'm importing/exporting, but for that to happen, it would have to be doing it in both SMD and DMX, and also somehow reimporting decompiled SMDs differently to exported ones, so that sounds like way too contrived a solution.

At some point later today (or maybe tomorrow), I will try Blender 2.78 (I'm currently on 2.79) and see what happens, and then try 2.77 if that doesn't work.

I will also get the mesh exported as DMX (and re-decompiled as a dodgy SMD) for you guys to take a look at. (At the very least, knowing whether other people can or can't replicate the error will tell me what I have to start looking at fixing).