I will not cover what each mask channel does, there is a great tutorial about it created by Valve. It can be found here or Downloaded on this link

This tutorial is about an easier way to work with all those textures to make you free to create and test the result quickly.

There is one information that is wrongly written on Valve's guide. On the page 6: "Tint Spec by Color - Mask 2 Channel 3 (B)".

You should consider the opposite colors on the mask, which means black represents the specular color using the base texture color and white being the default specular color defined on the vtm file

At this tutorial It will be also covered couple ways to test your shader without the need to reload your model in Dota 2 everytime.

For this tutorial we are going to use Quick Save maps

It is a Photoshop script created by Felix Schlitter that have some amazing features!! And it is for free. you can download or watch a video tutorial explaining here[felixschlitter.com]

Remember that if you use use it on an item for the Dota 2 workshop, to suggest the Valve developer to add him on the contributors list to be eleigible to receive that 5% extra share


After installing it, you can run using the script on the Action window or by pressing F3 (the installation will create this hotkey for you)

This is his interface with some maps already defined

For now I isolated the Texture maps area and marked the first things to define

• Layer name - This is the name of the layer group it will create. Use a name that will easily show for you what is the purpose of that group
  
• Convention - That is the name the script will add on your texture when it exports all your maps, I stronglly suggest to follow this so the Dota 2 compiler can recognizes it.
  
• Scale - That is the scale your final map will be exported. I usually works my texture 4 times bigger on each dimension; So if my texture is going to be 512 x 512 px I work my psd in 2048 x 2048 and let this field as "quarter"

As you may noticed, this script will create your layer groups in Photoshop and later export all your maps (saving you a lot of time)

Now the other options

• Format - pretty straight forward, I am using TGA right now. Pay attention that I have "Save Alpha" checkbox maker. If you dont export your alpha on the color map, the source will consider that is full black which means "Full transparent, and you dont want that
  
• Channel Replacement - One of the best features of this script. Notice that I have Color Alpha replacing the alpha. Color Alpha is a name of another layer groups i will create ,that means It will replace the alpha channel with the Color Alpha group
  
• Actions - There is folder icon with a green plus sign. That is a hotkey to create the group for me. I am going to click this one once and the other one in front of the Color Alpha on the Channel Replacement area

If you compare the names of the new folders, they are the same we had on the "Layer name" and on "Channel replacement". That is how the script will recognize it later to export. You can do whatever you want inside the folder but dont change the name. If you need to change don't forget to launch the script again and update the names.

The Normal Map configuration will be almost the same with just one difference on the alpha, It is not used by the engine so we are going to ignore it and make an smaller file

Don't forget to keep the "Convention" field like on the example

I already showed this normal map on the part 2 but I believe it would be usefull to post once more so it illustrates how an Normal map will look

Now starts the beauty of the Dota 2 shader. It uses each color channel of the next 2 maps (mask1 and mask_) to represent 8 different parameters.

Since the textures represents most of the size of the file on your asset, using one color texture to define 4 different parameters is great! It will save a lot of space on the game client and will reduce the amount of draw calls.

This is how a mask1 looks in the end




This is the setup for the Mask 1



The convention name is really important, I suggest you to use like that because Dota will look for that naming convention on the in game compiler

The "Channel replacement" fields are names used just for you, I like to keep smaller ones so they keep readable in photoshop without the need to change the layers panel's size.

Red channel

The detail map is used just in case you are blending a 5th texture for some animated effect like we have on Enigma or Razor. If you are not using just keep it black

This map will be replacing the red channel

the map is in gray scale now but dont worry if you have some colors on it, they script will use just the values of black and white


Green channel

This map is used to change how light affects your model based on the angle with the camera and the model face (fresnel). good rule for this one is to check if the base model of your character uses this one, if not I suggest to keep the same.

Another suggestion, if you are not used to what this one does, just leave it for later when you have everything working, this one doesn't visually affect too much compared to others

This map goes on the green channel


Blue channel

This ones is really important and should be used with care. The amount of white on this may change a lot the colors on your final object. A good behavior is to use it full black and add some gray details where you want to look more metal.
This map will affect the color and the specular. So have that in mind when using it and test every change!

It may show some small squares on the model, but with the animations this will not be visible.

On this example i have some gray on the metal bolts on the body straps, on the radiance sword (visible on the finished model on the workshop) and on the leg armor which was discarged after some interations.

Notice that I dont have a full white pixel, a full white with this mask means a black texture with a lot of specular.

You can achieve some nice and different results combining the metalness, tint specular by base color and the specular on mask_2.

This mask will help make your model to have more contrast by using different materials, and contrast is great to attract the eye in game.


Alpha channel

This is the Self Ilumination mask, By adding white on this one you are telling that those pixels will ignore in parts the "in game" lights.

It is good to attract attention to some details like runes or to the eyes but again, do not over do this one. It is interesting to have you model colors change from day to night and depending where they are in the map.

For example, when you are inside the Roshan pit your model with have a different light over it then on the lane. It helps giving you the mood of the location.

Notice that I have an small white dot on the bottom left corner of my texture. That is a good practice everytime you work with the mask_1. The reason is that if you go full black on this one, when compressing to vtf (valve texture format) the compression will see the black alpha and ignore it. Inside the game, when the engine loads your alpha and doesn't find it, it will be considered full white, and that will make your character full self illuminated.

By adding this small dota you prevent this by showing the compiler that you do have some pixel information on this alpha and it should not be ignored.

The mask 2 is the most time consuming of the 2 and also one of the most important to make your model stand out in game.

This is how a mask2 looks in the end




This is the setup for the Mask 2




Red channel

The specular is a really important map, it defines where in the character you are going to see the specular being reflected.
That is great to separate materials and help to show the superficial roughness of your model. Also really good if you want to show if something is dry or wet.
A good way to start it is by using your occlusion pass over a gray flat background. then you start to layer some materials over this flat gray.
Now it is pretty straightforward, Shinier parts are lighter then dull ones.

Notice my sword on the top left corner, almost full white, because it was a radiance sword, and I wanted it really shiny!


Green channel

the rim light is what will help pop your model from the background. It is a fake back light that will be based on the angle between the polygon normal and your camera.

Basically it will help outline your model creating a contrast with the bright light and the darker background.

Since the angle is calculated on all your model, you need to darken some parts where it is not interesting, like around the eyes and inside the mouth.
Usually the mouth will not have to much light bouncing inside it, that is one of the reasons you don't want the rim light to affect it.

Same for the eyes or other holes on your model, the hole may have some faces that will create a rim light, but it will look to fake.

I use to think like that: If I put a light behind my model, this part is going to be lighted or not? If the answer is "not" I paint it black.


Blue channel

This one means: Tint specular color with the base color.
This maps works on the opposite way, even the Valve's guide is confusing about it.

• Black = Full tint your specular with the base color
  
• White = No tint on your specular and it will use the color defined on the vmt.

The specular have just one color defined on your vmt file (material definition) and for different effects you may want the specular to be a little different.

That is really usefull for metals, if you let the specular being the same color of the overall model, your metal will look like a shiny plastic.

Same goes for the skin, you usually don't want your skin to have the default specular, unless it is a wet skin like Slardar's

Notice that my metal area, top left corner is full black, which means that my metal will have the specular tinted by the base color. On that case will be tinted by gold.


Alpha channel


The last map on our list. And it also affects specular.

This one will define the size of the specular on your model, really easy to use too.
Darker values will make your shine look more rough like a rubber
Brighter values will give your shiny something more to plastic or glass look

That is a lot of maps, you are working with 11 maps in total inside one PDS file. It is important to keep organized to keep a good workflow. Specially if you need to change something on your model weeks or months after "finishing" it.

If you use the script with the naming convention I suggested, your Photoshop layers will look like the image at the right.

On this setup you have the information of which mask you are working and a sum of what that does.


Remember the folder icon with the green plus sign we used earlier?

Just use them again following this order:

Also notice that I am not using all of them, I just need 11

Now that you have everything setup and all the layers working you can test it in game.

Actually I expect you already did this multiple times, you need to keep testing your texture to understand your changes and see if what you wanted is what happened in the game.

There is a lot of guessing when you start to work on a shader that depends on a lot of other factors, Your texture may be looking great on Photoshop, but not as good in game. That is why you test... everytime!

To export you textures you can hit the Action called Save All Maps or press shift + F2

The script will check your file to couple seconds



and then will start to export



After the export you will have the 4 files you need with your PSD...



... and with the correct naming for Dota 2 compiler

Working with Dota 2 shader may look a little confusing on the beginning but with time it get really easy to understand. It is just a matter of time

This way of organizing the shader by using the RGBA channels for each parameter is great to reduce the processing and the size of the asset.

Be sure to study other models in game to see how those maps are made. You can decompile the vtf files using a program called VTFEdit, it is for free and works great!
Here is the Valve wiki article about the program

Thanks for spending some time reading. If you find this article useful for you or someone else, please share the knowledge.

Spreading knowledge may not look an smart thing to do when you are competing to be noticed, but trust me, not everyone will return the favor, but some of them will, and it will worth!

For more of my work you guys can find at:

Portfolio

[www.tvidotto.com]

Facebook page

[www.facebook.com]