Render farm on Ubuntu

[Axle Grinder]

Hello everyone! This is one of many, many posts I've drafted for here; although most of the time I figure out eventually so I never post, but I lurk here often. I apologize for the awful structure here, but I'm super tired

Anyway, I was wondering if anyone has had any experience running SFM on Ubuntu Linux, through wine or another emulator. I know SFM isn't compatible with straight ubuntu. I have sfm running on windows on my main computer perfectly, so it's not going to be me animating through the Linux machine. My main computer is powerful enough and runs everything just fine, I can render on it, but I just want to set up a seperate render farm. It would make everything much faster and leave my main computer available to work on while rendering. Synching them would be a pain but doable. We all know what a pain in the ass rendering can be.

(Blender on the farm would also be great to do the image sequences, but that's an issue for another time)

So SFM gurus, what do you think? Would SFM run on a virtual machine? Would it render okay? Has anyone set it up on Ubuntu? Which emulator works best? Let me know anything if you can, and have a nice day.

[Zappy]

Originally posted by Axle_Grinder:

- We all know what a pain in the♥♥♥♥♥rendering can be. -

No, we don't. Care to clarify?

Originally posted by Axle_Grinder:

- So SFM gurus, what do you think? -

I think that Source Filmmaker uses DirectX, which is a Windows-only thing, for the graphics API thing, so you probably won't end up with 100% accurate results through emulation. I also think that Source Filmmaker already renders stuff so fast that you don't need a render farm for it (what might take Source Filmmaker seconds could take Blender quarters or hours, despite Blender's result not being that noticeably better (but still better), depending on the scene).

Yes, you could render something faster... but would the entire process be worth it, considering the amount of time needed to transfer assets and session files as well as the rendering possibly not being quite the same if not only done on Windows, compared to the amount of time it'd take to just render directly on your actual Windows computer?

(Do note that my opinion may be biased, due to me personally just using my single Windows 10 computer to work on and render Source Filmmaker stuff.)

[episoder]

well. as zappy said linux won't really work with directx. a windows machine is mandatory to get predictable results. the rendering itself can't even be spread across multiple virtual machines since it's a rather simple and fast hardware render.

in terms of secure pipelining and syncronizing, you gotta have sfm installed on the render machine. i'd guess you'd just need or should use a mod folder 'watch dog' tool something that automaticly duplicates and transfers new assets to the render machine or you gotta duplicate it by hand.

to trigger a render you may just load the session on the render machine and let it render it. you don't even need any shared account or anything steam. it's just a offline render.

i'm not sure if i can remember if there was a commandline tool that loads sfm without the ui, just for this style of rendering. i'm certain there was a thread for that or i'd not even think about it.

[Zappy]

Originally posted by episoder:

- the rendering itself can't even be spread across multiple virtual machines since it's a rather simple and fast hardware render. -

You could have each machine render a separate shot from each other, and have one machine render just the sound for the whole session in one go (as rendering the sound in multiple chunks probably leads to sound stutters/pops).

[episoder]

Originally posted by Zappy:

You could have each machine render a separate shot from each other, and have one machine render just the sound for the whole session in one go (as rendering the sound in multiple chunks probably leads to sound stutters/pops).

fair enough. i dunno tho how you would trigger aka setup multiple machines easily, to render those different shots. i'm not sure but, doing it virtual and efficient without messing up the hardware and memory you may need a graphics card for every instance of sfm. ofc you somehow gotta share the asset data without duplicating. i dunno how.

and as i posted i forgot if and how this commandline thingy works. this would have to be triggered in a console over a network connection from the master client. and it's gotta safely terminate. just a data rendering task for sure.

or you gotta let sfms run on the machines all the time and remote protocol the machine's sfms to load and unload with projects on demand. i dunno how this would be done tho.

[Shakes]

It works fine on Ubuntu (well, Mint at least but it's close enough). WINE runs it almost perfectly (viewports lag behind by a single mouse event for whatever reason). DirectX isn't an issue here since WINE does OpenGL translation almost perfectly for DX9. The performance difference between Windows and WINE is also basically null.

It runs fine but there is a parsing issue with how SFM handles terminators for workshop files. For whatever reason, some MDL files will have VMT data tacked on at the beginning of the file. If you're running through WINE I would suggest actually copying the data from your Windows install and ignoring workshop updates on your Linux install so that you don't have this problem in Ubuntu. You can remove the VMT data with tail but then it will think the model needs to be updated when you start SFM again.

Making a render farm is another beast to tackle, though it's not one I have experience with.

[Zappy]

Originally posted by Shakes:

- DirectX isn't an issue here since WINE does OpenGL translation almost perfectly for DX9. -

The keyword here is "almost". Regardless of how close it gets, it'll never be quite perfect.

I'm not saying that there will be abnormally huge issues not present on Windows, but I'll be very surprised if there aren't at least a few differences that can be seen in a side-by-side comparison between an emulated-Windows-on-Ubuntu render and an actual Windows render. (Whether or not you'd have to use image-analysis tools, I don't know, but still, any difference between the two, however small it may be, is still a difference.)

[episoder]

Originally posted by Shakes:

some MDL files will have VMT data tacked on at the beginning of the file.

that's one odd error. how can that happen? is that when you download the assets? then i'd guess the steam packet delivery unpacker does not read the compressed packet structure correct. my guess, it's a steam filesystem emulation issue.

[Shakes]

Originally posted by episoder:

Originally posted by Shakes:

some MDL files will have VMT data tacked on at the beginning of the file.

that's one odd error. how can that happen? is that when you download the assets? then i'd guess the steam packet delivery unpacker does not read the compressed packet structure correct. my guess, it's a steam filesystem emulation issue.

Yeah, it's very weird. Every other Source game works fine through WINE, including workshop downloads, except SFM. I have a theory that it has to do with how Steam packs workshop downloads into a single contiguous file before unpacking them. If the VMT isn't properly terminated (i.e., odd number of curly braces or quotes) then the EOF is interpreted as being somewhere else and WINE unpacks it improperly, thinking the VMT and MDL are the same file.

[⚡🍉OverlordTomala🍉⚡]

I tested SFM in Proton recently under Mint 18.3, though I didn't look too far into workshop files (though I already cheated and copied myWindows SFM install). One thing that holds me back from going any further is being stuck with exporting in AVI or image sequences. I haven't figured out how make it see Quicktime Alternative. If I could figure that out I'd like to test by doing a small project while running it under Proton just to see how stable it is.

[Zappy]

Originally posted by ⚡🍉OverlordTomala🍉⚡:

- One thing that holds me back from going any further is being stuck with exporting in AVI or image sequences. -

How does this hold you back? If you are capable of doing an image sequence export, then all is well, as that's the only "video" type export (other than "Sound" if you count that as a separate export type) that you even should be doing (especially on Windows, but still also outside of Windows).

[Marco Skoll]

Originally posted by ⚡🍉OverlordTomala🍉⚡:

One thing that holds me back from going any further is being stuck with exporting in AVI or image sequences.

You really *should* want to render in image sequences.

Although non-Windows versions of Quicktime probably don't have the same security issue they do on Windows, you will still suffer from the same poor quality that SFM uses when trying to encode through Quicktime and, very importantly, the "attempting to encode one file over hours" problem.

If SFM crashes mid-render, then if you're writing into one single video file, you end up with an incomplete, corrupted file.
If you're writing into separate image files, then you still have every previously rendered frame saved intact to disk and can restart mid-render.

Given that running SFM in a non-Windows environment is very likely going to make the program more unstable than it already is, you should really want to capitalise on that ability to recover crashed renders.

Encoding the frames into a video at the end really isn't very complicated and takes only a couple of moments (certainly compared to the time it originally took to render), will give you much more control over encoding quality, and because it takes much less time, the encode is less likely to be corrupted by a crash.
It really will save you headaches in the long run.

[⚡🍉OverlordTomala🍉⚡]

Originally posted by Marco Skoll:

You really *should* want to render in image sequences.

Although non-Windows versions of Quicktime probably don't have the same security issue they do on Windows, you will still suffer from the same poor quality that SFM uses when trying to encode through Quicktime and, very importantly, the "attempting to encode one file over hours" problem.

I don't use Quicktime under Windows per se, but Quicktime Alternative instead. I tried that and regular Quicktime under Mint but to no avail. As far as image sequences vs video containers its nice to do a quick test render to see how my animation looks as a "final product", as that is faster than waiting for an image sequence to complete and finding one thing that I want to change, then having to re-render.

It is nice to have options, and see how well they perform when comparing side by side to Windows.

[Zappy]

Originally posted by ⚡🍉OverlordTomala🍉⚡:

- As far as image sequences vs video containers its nice to do a quick test render to see how my animation looks as a "final product", as that is faster than waiting for an image sequence to complete and finding one thing that I want to change, then having to re-render. -

Actually, that specific case is a case where image sequences are much faster to deal with than videos, in fact. Keep in mind that an image sequence will render faster than a video in Source Filmmaker, so the extra time to convert the images into a video kind of makes it on-par with each other rather than take longer. But then if you notice one thing that you want to change, then with an image sequence render, you just have to re-export the shot(s) where you change something and then convert the image sequence into a video again, whereas if you were to render a video from Source Filmmaker, you would have to re-export the whole session, which takes much longer (unless the thing that you want to change is throughout all shots, in which case the two exports' durations will still be on-par).

[liu.wanfang]

I hope SFM support proton.

[Shakes]

Originally posted by liu.wanfang:

I hope SFM support proton.

Not sure if you can install it from the native client directly, but you can import your WINE Steam library from within the native Linux client and SFM will show up under "Software" if you have it installed there. You won't have your saved settings or recent history though, since Proton will run it from its own isolated wineprefix.