Making cartoon/comic style posters

So I've been experimenting with the ways you can do cel-shaded posters in SFM, and I think I developed a solid SFM artwork pipeline that is easy enough to follow (given that you are quite familiar with both SFM and your photo-editing software). It's nothing too complex really, but there doesn't seem to be many cel-shading guides out there, so I decided to make my own.

Who is this guide for

You should check if the statements below are applicable to you:

• "I have a general understanding of Source's material system. I know what phong, bump map, and env map are and how to work with them, etc."
  
• "I have a decent photo editing software that can make selections by color channel and\or color range (e.g. Photoshop, Affinity Photo)"
  
• "I know how to make lighting based on high-radius (overwritten to 100+) lights"
  
• "I feel confident using both SFM and my editing software (I know how to override materials, pick right render settings; am familiar with layer system and general functionality of my photo-editing software, etc.)"

Ideally, all of them should be "Definitely true". If not, keep in mind I won't be getting in every little detail in this guide, but I'll try to keep it as easy to follow as possible

Using a tablet and having some drawing experience helps too

Small disclamer

I'm not a 2D artist, so all the things I do here are based on observation and techniques commonly used in Blender. My only rule here is "If it looks pretty, then it works. If it doesn't, then it doesn't work.", don't expect a lot of drawing theory being used here, lol.

With that covered, prepare yourself. It's gonna be a long read...

The main idea is to exclude just enough information and realism from the rendered image so only the most important details are left, achieving the illusion of it being drawn.
Let me explain:
It's easy for a computer to render a perfect sphere with physically-correct shading, but it's hard for it to make an impression of it being drawn on paper. A paper drawing differs itself from a photo or a 3D render by it's imperfections.
It also doesn't take much for our eyes to define an object. That's why by looking at this sketch:
(sketch credit)[www.deviantart.com]
..we see a human figure, instead of a bunch of weirdly shaped lines.

It may seem obvious, but nevertheless it's important to keep in mind.

A 3D toon artwork is usually defined by the usage of:

• Cel-shading
  
• Outlines
  
• Cartoon or drawn-like character proportions

Naturally, these aspects are not consistent in every piece of art, but we'll be mainly focusing on them here.

Cel-shading

Cel shading is basically a stepped transition from highlight to shadow. It's a common shading technique in drawing:


You should be familiar with it since TF2 also uses cel-shading on most of its models and I feel like 99% of the SFM userbase worked with the TF2 assets at least once :)
The difference is especially noticeable on the arms.

Outlines

This one should be easy: it helps differentiate the object from the background. AO-based outlines in SFM are pretty garbage, but it's not like we have any alternatives aside from drawing them manually in post


Random trivia:
There are 2 methods of doing outlines in 3D (at least these are the ones I'm aware of lol): adding outlines after rendering the image based on the normals information:
(Image credit: answer from KTC on gamedev.stackexchange[gamedev.stackexchange.com])
...and using so-called "inverted hull method".
The first one is pretty self-explanatory, so let me talk about the second one. It's done by extruding vertices of the mesh along their normal directions, inverting the direction of the resulting faces, and making them pure black (or the other color you need). So you're rendering outlines alongside the model.

Here's my port of Dust from Lethal League Blaze as an example:

If you look carefully, you can see that the outline is essentially a thicker copy of the model with a pure-black color material.

Dunno which one of these SFM uses, but I'm guessing the first one since the outlines in SFM are all the same width regardless of the distance of a model from the camera (click on the images to zoom in to see the difference):

(SFM outlines vs inverted hull method outlines baked in the model)

Cartoon-ish characters proportions and styling

The point I'm trying to make here is that the character you want to make a render of should be created with the cartoon art style in mind. Cel-shading on TF2's mercs looks good because they were created with the toon style in mind: they have exaggerated features and textures that were created according to the strict art guidelines[polycount.com] that focus on simplicity. A super-cartoony shader on a realistic model (say, Gman from Half-Life: Alyx) wouldn't necessarily look bad, but kinda weird at the very least.
So I'd recommend hiding all bump maps and env maps on your materials before rendering. They create additional detail that we don't need in our case.
Here's an example of disabling fake pbr and normal mapping:

You can also try repainting the textures to be more cartoony. TF2's texturing guidelines work great with cel-shading, so you can use them as a reference.

Pros

• Quick and easy
  
• Good for animation

Cons

• Works only with light_dynamic (map lighting)
  
• You don't get shadows
  
• Can't change lights color, direction, or any other property without postprocessing or recompiling the map

This one is easy: all you need is a map with lighting and a sharp light warp texture, and you're good to go:


$lightwarptexture:


If the map has multiple lights it may create these stepped shades:


Usually these aren't very appealing, so if you want to change that you need to
a. Find a different place on your map with more preferable lighting
b. Open up Hammer and make a custom map with a lighting setup you want...
That's the main problem of this method: you're very limited in terms of flexibility of your lighting.
I've made a custom void map with only 1 environmental lighting, and it gives these shadows:

Here it is btw: drive.google.com/file/d/1dx0ppEt2bG83D9Mx589bB84NyVqK0GPb/view?usp=sharing

There's really nothing else to say about this. I think this is more useful for rendering animations, it's kinda too limiting for a proper poster.

I'll be honest, the core concept here is the same as in this guide, but we're going to take it to the next level.
Pros

• Full control of your lights
  
• Has shadows
  
• You can easily paint in shadows or highlights later during postprocessing

Cons

• You'll need to make up to 6 renders.
  
• You can't really preview the result until you start composing
  
• It's possible to render an animation using this method, but honestly, it's better to learn a bit of shading in Blender and render using Eevee rather than make 5 or 6 renders of your animation.

Here are a few examples:
https://steamcommunity.com/sharedfiles/filedetails/?id=2113387316
https://steamcommunity.com/sharedfiles/filedetails/?id=2116910268
https://steamcommunity.com/sharedfiles/filedetails/?id=2118584233
I'll be making the latter one here.

The idea is to compose the final image from 4 layers: the highlight layer, the mid-tones layer, the rim-light layer, and the shadow layer. We'll also need to create 3 masks to define edges of the layers mentioned above (except for the shadow one, it's going to be on the bottom of the layer stack). After combining these layers it's gonna be you regular poster post-processing: color-correcting, making effects, adding bloom, etc etc etc.

First things first, we'll need to create 4 lighting sets.

Don't forget to disable any AO you have.

1. Masks for cel-shading

We need to manually mix highlight, mid-tones, and rim lights in the photo-editing software (I'll be using Photoshop). To do that we need 3 masks. We're going to make them using extremely bright lights.
All of my lights below have:
intensity: 1000000 (you need overwrite light's intensity max value)
constantAttenuation: 1
shadowAtten: 0
shadowDepthBias: 0

Highlight mask	Mid-tones mask	Rim light mask
shadowFilterSize: 0 radius : ~10 (note the yellow eyebrow: we'll need to fix it later in post. We want all of our masks to be grayscale)	shadowFilterSize: 0 radius : ~80 It's practically a copy of the highlight mask, except with much heigher radius.	shadowFilterSize: ~0.05 radius : 0

Naturally, you don't have to make your lights exactly the same as here. Experiment with radius and shadow size!

All unmasked space is going to be shadowed.
Note that materials with env mapping, rim lighting, or emissive properties can result in inaccurate masks!!

We can reduce render times and save some hard drive space by making these masks pure red, pure green, and pure blue, and rendering them in one go:

In Photoshop, by holding ctrl and left-clicking on one of the channels in the "Channels" window you can select all of the white areas in it. We can use that selection later as a mask.

2. Base lighting layer

This is going to be on the bottom of our layer stack. Just make your regular soft lighting setup using high-radius lights. Take positions of your masking lights into account while doing that: place your key lights where highlight mask is and rim lights where rim mask is.


I won't get into detail on the lights here, check out this guide if you're having trouble with lighting:
https://steamcommunity.com/sharedfiles/filedetails/?id=1684499173
In short, there are 2 key lights with sharpest shadows and radius set to 150, 1 ambient light above with radius set to ~380, the same one below with color changed to act as lighting bouncing from the floor, 2 fill lights with disabled shadows and radius set to 400 and 2 rim lights with soft shadows with the radius set to 75.

3. Highlights and mid-tones layer

We're going to use our highlight and mid-tone masks on this layer. Duplicate the base layer lights and make the highlight wider:

I disabled rim lights to keep precious shadowed light cap low since they'll get masked out anyway.

4. Rim lighting layer

We're going to use the rim light mask on this one. Once again, copy your base lighting setup, but this time make your rim light bigger:
I disabled highlights for the same reasons mentioned before.

Misc renders

You can also render the outline layer. Outline width in SFM is sadly fixed, so the only way to change it would be to later during post-processing select the outline, expand the selection, fill it in with black, and then do some clean-up.

I've also made a mask to cut the character from the background:

See this guide to learn how to do it:
https://steamcommunity.com/sharedfiles/filedetails/?id=821085748

So now you should have 6 renders:


Let's quickly compose them and see how it turns out! Use character mask to remove the background on all of your renders, make a copy of the key lighting layer, and apply masks from the first step to their respectful renders. Make the key light layer the top one, set its opacity to something like 70%, mid-tones layer should follow it with opacity ~50%, then the rim light layer with full opacity, and finally the base layer. Adjust the opacity of your layers to your liking. If you have the outline layer, place it above everything else, and set the blending mode to "Multiply".

Alright, not too shabby. At this stage, it's important to note all of the problems to fix later during post-processing or by re-rendering some of the layers.
Here I feel like the highlight is overexposed and some of the layer edges are kinda ripped. But luckily the first thing can be fixed during color-correction and the second one is only a matter of adjusting the mask of the needed layer slightly, so I'll continue from here.
Also, it looks like the problem I had with an eyebrow resolved itself, it seems like the remaining color information there was covered by the second light with an intensity of 1000000.

Smoothing edges

So there's a really cool feature in Photoshop for this kind of stuff called "Select and Mask". It lets you fine-tune the sharpness of your selection, which is exactly what we need:

Naturally, it's not perfect, but it can save you a ton of time by smoothing all the small-scale roughness

Since we use masks, it's easy to manually fix troubling edges and paint in some shadows or highlights. I recommend using a drawing tablet if you have one.
(I can't draw so that's all I can do :'D)

Outline

Outline in SFM is kinda shallow, so usually I change levels to bring out the blacks more

I'm also gonna make the outline a bit thicker by selecting it, expanding the selection, and filling it in with black, plus some clean-up afterwards:

Outline coloring

If you can't decide whether to keep the outline or not, you can try tinting it with the near-by colors of the character:

Color correction, bloom, and final touches

I've always done color correction by an eye, dunno any rules :D. My advice - play with it until it looks pretty to you. Eventually you'll get the hang of it.

If you're using Photoshop, I recommend color correcting with the Camera Raw filter





It's very easy to make bloom during post-processing: create a new layer, color pick the brightest parts of your image, and draw onto them using hard brush. Then blur this layer, adjust it's opacity to your liking, and you're done! Be careful and don't overdo it tho


We're almost done, so it's time to do some final touch-ups. Here I fixed minor clipping issues, fixed some mask edges, tinted sleeves with a goldish color, and brightened eyes a bit.
The other cool thing you can do is add a layer of dots to give the whole thing a vibe of comic book art. Select your character on the base layer, and exclude areas of your selection that collide with the highlight, mid-tone, and rim light masks. This selection should cover all shadows on your character. Now feather the selection and use it as a mask for your dots layer, which should be right below the outline layer.
It's very easy to do in photoshop: ctrl+LMB on the preview of your base layer and ctrl+alt+LBM on the masks of your other layers in the stack. Here's a ♥♥♥♥♥♥ gif to help you out:
https://steamcommunity.com/sharedfiles/filedetails/?id=2118541217
Here's a 2k image of dots if you're too lazy to make one yourself :)



I didn't have a specific background in mind for this poster, so I went with a simple one (also because I've been doing this for 2 days and I'm already getting tired of this ♥♥♥♥)

AND WE'RE DONE!

Random tip:
If you have banding issues (visible stripes of color instead of gradients that supposed to be smooth), try making a noise layer, setting it to "Overlay" and lowering the opacity to something like 10%. This should help get rid of banding, but it's gonna make your final image take up more hard drive space.

Has banding

Has banding reduced

Appreciate if you made it all the way through. It's kinda sad that this kind of style is rarely explored in SFM posters, so I hope this guide will help people to start using it more often.

If you find any mistakes, please mention them in the comments, thank you!

Also pls rate the poster from this guide on the artwork page :D
https://steamcommunity.com/sharedfiles/filedetails/?id=2118584233

[Mello]

i aint readin allat

[CalbDoesStuff]

What if i wanted to make it an animation?

[✧ Hat Kid Poptarts ♥]

man, just by reading the statement i knew ill never be that good lol

[Big Glock]

I cant seem to increase the radius even after changing the min max and default it still stays at 1

[skullmxde]

nice

[misteremre]

good guide

[Pretzel]

Great guide dude!

[Prof. Purble]

I've wanted to experiment with this art style for awhile now but there was never really a decent guide for it. Thank you very, very, VERY much for this. I will put it to good use.

[EL BOLAS]

so nice

[Leg4Me]

very good