How to cel shade your SFM posters

Cel shading is a well-known shading trick that can heavily make your posters stand out visually. In this guide I'll try my best to demonstrate how you can apply it to your own Source Filmmaker renders.

Before I begin

None of what you will read here belongs to me. This guide is just a rewrite of SK_Form's cel shading tutorial. I have simply taken his permission to create this guide. If you have encountered any problems, either check out the video, or contact SK himself. Thanks and I hope this guide proves helpful to you.

So, what is cel shading? (Feel free to skip)

Okuha describes cel shading as:

Автор сообщения: okuha.com

an art style of non-photorealistic rendering designed to make 3D computer graphics or artworks look three dimensional by creating flat colors on top of a base color, making the object look three-dimensional while still keeping that 2D effect in it.

It's heavily used in many anime-based videogames and the recent The Legend of Zelda titles and it helps a lot at making their artstyles stand out, and I'm gonna try to showcase how to execute that type of shading on your SFM renders in this guide. So, strap in!

Requirements

Besides SFM itself, you will need a photo-editing software. It doesn't matter what software, as long as it supports blending types and hue/saturation effects.
I'll be using Photoshop for this, but free softwares like GIMP should do just fine as well.

Rendering

For this tutorial, I'll be showcasing the steps towards making this render of Shota Aizawa from My Hero Academia blepping and use it as an example.
Don't ask why I made him blep of all things. I get a kick out of being random with my art.

Before you fully begin, make sure ambient occlusion is disabled. Simply right click on the viewport, select "Render settings" and tick off "Ambient Occlusion".

Now, start posing your character like any ordinary render. Avoid putting anything in the background, since it'll mean you'll just have to shade the background as well, which is just unnecessary work for what we wanna do here.
Also, try posing the render in a map without lights. black_void.bsp should do just fine, but this can also come in handy:
https://steamcommunity.com/sharedfiles/filedetails/?id=2643240491

After you're done with the posing, you need to give the render a fully white background. You can use this Workshop asset (the white screen is stored as a skin. Should be skin 3).
https://steamcommunity.com/sharedfiles/filedetails/?id=186065555

Also, make sure to create an animationset for your camera by clicking the + button above the animationset editor, selecting "Create New Animationset For Existing Elements", and selecting the camera. We need this animationset so we can disable the bloom. Just slide bloomScale all the way to the left.
























So this is what we have now:

We need to make 4 renders of this pose:

Render 1: The base

Proceed to render the image. Either as a poster, or shorten the sequence's length to one frame through the Clip Editor and render the sequence as an image sequence. You do you. We'll name it 'base'.

Render 2: The outline

Go ahead and turn Ambient Occlusion back on. But make sure you set Ambient Occlusion Mode to 'Outline Only'.

So you have this now:

While it looks fine enough from here, once we get to mess around with it on our photo-editor, you will see that it can be way better. Render this outlined pose as well and name it 'outline'.

Render 3: Shadows

Turn off ambient occlusion once more, enable your lighting and spawn in a light. You need to alter a few slider settings:
Put shadowAtten all the way to the left, set shadowFilterSize to 0.01 (double-click on the slider to write its value manually), and pull constantAttenuation and linearAttenuation all the way to max value.
Now, you need to go to your console, and type it this command:

mat_fullbright 2

This command basically removes all color and makes the render a black-and-white image.
Now go ahead and position the light. The black parts are going to be our shadows in the final render and the white parts will be erased.

Render this too. We'll be naming it 'shadow'.
If you are not 100% happy with how the shadow turned out, don't worry. We'll be fixing them up on our photo-editor.

Render 4: The mask

This one's simple, just delete the light and export the black and white image you'll get. Name it 'mask'.
This isn't fully needed. It's just there for when we want to erase the white background once the shading is done without potentially messing something up, so if you know of any other reliable methods for removing backgrounds, you can skip the mask.

Now we've finished up with the rendering, we can hop onto the photo-editor.

Applying the cel shade

Import your renders in this order of layers:






Now, having the shadow layer selected, turn the blending mode to 'Multiply'.

































It kinda looks like a character mugshot from Shadow Fight 2 now.

Change up the opacity to your liking. I'll be going with 30% for now (I ended up altering it several times later on, hence why the shadows look darker in the next few screenshots).

Now, once you zoom in, you'll start spotting problems with the outline, like several parts straight-up missing and it looking a bit too pixelated.

First, having the outline layer selected, click on 'Filter' and select 'Filter Gallery'.



























Go to 'Sketch' and select 'Photocopy'.
Adjust the darkness and detail sliders to your liking and apply the filter.
Note that I have gotten a black background here with white outline, because I didn't choose my foreground and background colors properly (they're meant to be black and white respectively). While you are supposed to have a black outline and a white background. If you ended up making the same mistake I did here, simply press Ctrl+I to invert the layer's colors.

Now, set the blending mode to 'Multiply' like the shadow layer.

Now, it looks a bit better and pops more, but it's still too pixelated and has parts missing. So we're going to fix that. (This is the time-consuming part. Grind your teeth, drink some water, and put on some music, because this'll take a while)

Go ahead and select the pen tool (or press 'P' on the keyboard).

A brief summary of the pen tool (since it's crucial to understand it for this part)

The pen tool basically allows you to create paths by connecting nodes together. You can create a node by a simple left-click, and curve the path between two nodes by holding the left-button when you create the node. You can always hold Ctrl and click and hold on a node to alter its position if you messed it up and also press Ctrl+Z to undo anything you've done with a node like any ordinary action within the software.
You can turn paths into drawn lines by right-clicking and selecting "Stroke Path" while the pen tool is equipped. Make sure the tool is set to 'Brush' and click Ok. The lines' size and sardness will be determined by the Size and Hardness settings of your Brush tool.
And after that, you can delete paths by right-clicking again and selecting 'Delete Path'.
Basically, we want to create a new layer on top of the outline layer and make paths on the entire outline of the render and turn them into lines. This outline that we have just created through Filter Gallery is solely meant to be a tracing reference for the actual outline in the final render. (How many times did I just say 'Outline'??)
(The red arrow is a line drawn with the pen tool and the blue one is from the exported outline from SFM)
As you can see here, the drawn line is way smoother and looks nicer than the one SFM itself has exported.
Now that we have drawn our own outline, we can move on to the shadows.

There are a few things you can do here:
Either leave the shadows as is.
Or fix some spots where the shadows don't exactly look right. (Which is what I'm aiming to do here)
Now, remember how this shadow layer even works here. It's basically a black and white image where the white has become transparent through the blending mode and the black serves as the shadow.
In other words, by simply coloring an area black with the brush tool on the shadow layer, you create a shadow and you will be removing the shadow by coloring said areas white.
There are two ways to select certain areas to color them over:
* The lasso tool
* Or creating a shape using the pen tool (which should've already become you best frenemy by now) and clicking on 'Selection'.

Now we can move on to coloring the shadows.

Coloring shadows: The easy way

While you are fully free to just leave your shadows grey, coloring them certainly gives them a much needed visual flair.
Select your shadow layer and add a Hue/Saturation layer on top of it.
Make sure you right-click on the newly made layer and select 'Create Clipping Mask'. This prevents the effect from affecting all layers below it and only affects the shadow layer.
Now in the effects window, tick 'Colorize', bring Brightness and Saturation sliders to the right and then color the shadows with whatever color that suits you with the Hue slider.

Coloring shadows: The ever-so-slightly-but-not-really-less-easy way

This basically requires you to seperate the shadows into seperate frames. Simply duplicate the shadow layer, erase the parts you want to give a different color to on the duplicate, and erase the other parts on the main layer.
Now you can go ahead and add the same Hue/Saturation layers to the shadows layers like the previous method.

Congratulations, your cel shading is basically done. Now we just gotta work out our background and our render is good to go!

Background and post-processing

Let's remove the background first.
This is finally where the mask we rendered gets to shine.
Import the mask on top of everything and go to 'select' and click on 'Color Range'.
Click on the white to select it as the color and leave the newly opened window with these settings:
Now the entire white background should be selected.
Now, turn the mask layer invisible, go to the base layer and hit the delete button on your keyboard.
The background should be gone now.

From here on out, you're really on your own with what you want to do in terms of background. You can either crop down the already transparent render and call it a day, or put some random image behind the base layer and apply some blur on it, or do your own scenebuild within SFM and throw the exported result as the background. It's all on you.
But for all intents and purposes, you are officially done. Thank you for reading the guide and once again, all credit goes to SK for his tutorial.

Addition notes

* Certain models (tf_movies models for example) have an odd shiny effect on them even when the lighting is disabled. This shine is caused by somethung within the Source engine named 'Phong'.
I am not qualified enough to fully explain how it works, but in order to disable it, simply type thus on the console:

mat_phong 0

Keep in mind, however, this command obliterates all ambient occlusion, even if you have it on through Render Settings, so render your outline before running this command. (or simply run

mat_phong 1

to turn the phong back on)
* constantAttenuation and linearAttenuation aren't required to be maxed out for the light. Simply running

mat_fullbright

after spawning the light should do the work.

Credits

This guide and the video wouldn't have possible without the people within SFM discord, especially kvaran and MajinBro, these guys explained the steps in an extremely basic way to which SKForm composed in his video, there are its and bits that haven't been mentioned that we're too complex for the tutorial however they won't hesitate to help out anyone getting into it, join the SFM community discord server for their assistance!

[tyebgat]

mat_fullbright 2 is just erasing the textures on all models, not turning the render balck and white for me

[mountaindewdz]

I did the full mat bright and it didn't go full black and white, there was still some color left on my characters arms

[antho p]

ok so why is my thing after i finish and erase the white background there is this many black spot is this because i did not use the mask on gimp

[Kooshλ Kλrimi]

I'm not too experienced on Gimp but I'm fairly certain it has its own equivalent of PS' pen tool

[Kooshλ Kλrimi]

It's because filter gallery is a unique thing to Photoshop that Gimp doesn't have (my copy is totally legit btw just figured I specify I totally own PS legitimately :3)
Tbh you really don't have to use the filters. You can just draw the outlines on the flat render all by yourself because the "sketch" that the filter generates is a mere reference

[antho p]

also how can you do the sketch in gimp too

[antho p]

guy for filter gallery why can't i see it on the filter for gimp

[Kooshλ Kλrimi]

@Oscar As long as the software supports blending modes and hue/saturation (or smth similar to it) it'll work. The way you draw the lines/shadows is ultimately up to you

[Oscarknight]

Can this work with Autodesk sketchbook?

[giggleshitter1945]

sandvich good!