Moussa Dembélé – www.mousman.com- mousman@hotmail.com
Cel shading – aka toon shading – is used to give a cartoon style to a 3D model.
In this article I explain a way to do this with AGAL.
Sorry, either Adobe flash is not installed or you do not have it enabled
A simple shader using a Lambert factor for lightning will give you somthing like this :
So how to get to a cartoon effect starting from this shader :
The cell shading has two parts.
The first one is the outline of the model.
Outline :
They are different ways to do it.
The one I’m using is based on the single pass cel shading given by Jean-marc Leroux (Minko).
Most of the methods require several passes in order to make the outline.
This one has the advantage to be a single pass method.
The result is not perfect though and to get a better rendering you should go for a multi passes outlines methods.
but still this method does its job.
The principle is to calculate the dot product (cosinus) between the camera and the vertices normal.
A negative value indicates a vertex on the model’s backside.
If so, we move the vertex toward it’s normal in the vertex program, pass a backside vertex indicator to the fragment program,
and in the fragment program we color this vertex in black.
The model will be surrounded by black vertices and the amount of vertices shift will set the outline thickness.
Easy, isn’t it ?
I’ve modified the method a little bit.
The first modification is due to the fact that to move the vertex, we multiply its normal vector by a constant and add this result to its position.
By doing this we will have a thickness which is dependent on the perspective.
It can be a good thing, but in my case I wanted the outline thickness to be constant everywhere on the model.
(But still be abble to modify the thickness for the whole model, in order for instance to change it according to the main z position of the model).
This is pretty simple to do, as the perspective computation involves a division by z.
So we just have to multiply the vertex displacement by z.
The second modification is due the fact that with some models the rendering is not so well,
this is a downside of this method (it ususally occurs with sharp edges, and with thick outlines).
I’v tried to get a better result but this result is just slightly better (depending on models)…
as you can see below (original method on left, modification on right)
Still better than nothing…To do that I recalculate the Lambert term in the fragment shader,
and in addition to setting to black backside vertices, I also set to black vertices near the edge on the frontside.
That’s it.
Another downside is a flickering effect when moving with thick outlines.
I couldn’t solved this and if you have a solution please share 
You can see an exemple below :
Roll over to activate
Sorry, either Adobe flash is not installed or you do not have it enabled
The second part in cel shading process is about shadows and colors.
quantized colors :
We don’t want a smooth gradient in the lightning of our model.
What we want is a cartoon style and in this style the number of colors is limited.
To do that we can use a texture map, the advantage of this method is that this texture can be made by a graphist,
and easily adjusted.
But I’m going to follow another way : calculation of the quantized colors.
And in fact it’s not really the colors (as the model has few colors), but the lightning which will be quantized.
The Lambert term gives us a smooth gradient of shadow (from 0 to 1),
starting from that we set the number of colors we want (3 in my case) and split the gradient in 3 parts.
From 0 to 0.33 the value will be 0, from 0.34 to 0.66 the value will be 0.33 and from 0.67 to 0.99 the value will be 0.6.
For that we just have to multiply the lambert term by the number of colors, take the integer part and then divide by the number of colors.
easy ?
I’ve modified this method because I wanted the shadow strips to be nearer the edges.
I combine the Lambert term of the lightning with the Lambert term from camera.
By doing that there will be strips around all edges and not only around the ones that are not in plain light. It will looks more like a hand drawed model.
To get the strip nearer to edges I multiply by 2 the Lambert term of the camera and I’m also rounding the result of multiplication of the Lambert term by the number of colors before the division.
You can see the result below, on the left without modification, on the right with modification :
So now we mutliply the model colors by the values calculated and we have the quantized colors.
Then we just have to combine the outline with that and……. it’s done !!
See the complete shader class
package {
import com.adobe.utils.AGALMiniAssembler;
import flash.display3D.Context3D;
import flash.display3D.Context3DProgramType;
import flash.display3D.Context3DVertexBufferFormat;
import flash.display3D.Program3D;
import flash.display3D.VertexBuffer3D;
/**
* @author Moussa Dembélé (www.mousman.com)
* va0 : position x,y,z
* va1 : color r,g,b
* va2 : normales
*
* vc0 : scene matrix
* vc4 : object matrix
* vc8 : object matrix without translation (only rotation)
* vc12 : camera position
* vc13 : cel-shading data
*
* fc0 : ambiant light
* fc1 : light diffuse color
* fc2 : light position
* fc3 : material ambiant color
* fc4 : cel shading data
*/
public class MCelShader implements IMShader
{
private var _vertexProgram:String;
private var _fragmentProgram:String;
private var _shaderProgram:Program3D;
private var _vertexShaderAssembler:AGALMiniAssembler;
private var _fragmentShaderAssembler:AGALMiniAssembler;
private var _dataPerVertex:int = 7;
public function MCelShader()
{
createVertexProgram();
createFragmentProgram();
}
//----------------------------------------------------------------------------------------------
//----------------------------------------------------------------------------------------------
private function createVertexProgram():void {
_vertexProgram =
"m44 vt0 ,va0, vc4 \n" + // vt0, object position in the scene
"m44 vt1 ,va2, vc8 \n" + // vt1, normal position in the scene
"mov v0 ,va1 \n" + // vertex r,g,b
"mov v1 ,vt1 \n" + // v1, interpolate normal position in the scene
"mov v2 ,vt0 \n" + // v2, interpolate object position in the scene
// outline
"sub vt2 ,vc12, vt0 \n" + // get dir from the vertex to camera
"nrm vt2.xyz ,vt2 \n" +
"mov vt2.w ,vc13.w \n" +
"mov v3 ,vt2 \n" +
"dp3 vt1 ,vt1, vt2 \n" + // dot product (normal, position)
"slt vt2 ,vt1, vc13.x \n" + // on edge ?
"mov v4 ,vt2 \n" +
"mul vt3 ,va2, vt2 \n" + // move vertex towards normal
"mul vt3 ,vt3, vc13.y \n" + // thickness
// compensation of the perspective
"mul vt3 ,vt3, vt0.z \n" +
"add vt3 ,vt3, va0 \n" +
"mov vt3.w ,vc13.w \n" +
"m44 op ,vt3, vc0 \n" ; // transform vertex x,y,z
}
//----------------------------------------------------------------------------------------------
//----------------------------------------------------------------------------------------------
private function createFragmentProgram():void {
_fragmentProgram = "" +
"sub ft0 ,fc2, v2 \n" + // get dir from the fragment to the light
"nrm ft0.xyz ,ft0.xyz \n" + // normalize the light direction
"mov ft0.w ,fc2.w \n" +
"mov ft1 ,v1 \n" + //normals
"nrm ft1.xyz ,ft1 \n" +
"mov ft1.w ,fc2.w \n" +
"mov ft5 ,v3 \n" + // camera direction
"nrm ft5.xyz ,ft5 \n" +
"mov ft5.w ,fc2.w \n" +
//calculate diffuse term - ft3
"dp3 ft3 ,ft1, ft0 \n" + // light
"dp3 ft7 ,ft1, ft5 \n" + // camera
"add ft5 ,ft7, ft7 \n" + // double the dot product
"mul ft3 ,ft3, ft5 \n" + // combine light and camera
"sat ft3 ,ft3 \n" + //
// cel-shading
"mul ft3.xyz ,ft3.xyz, fc4.x \n" + //
"add ft3.xyz ,ft3.xyz, fc4.w \n" + //
"frc ft4.xyz ,ft3.xyz \n" + //
"sub ft4.xyz ,ft3.xyz, ft4.xyz \n" + //
"div ft3.xyz ,ft4.xyz, fc4.x \n" + //
// diffuse light
"mul ft3.xyz ,ft3.xyz, fc1.w \n" + // multiply projection with light's diffuse amount
"mul ft3.xyz ,ft3.xyz fc1.xyz \n" + // multiply projection of direction to light on normal with light's diffuse color
"mul ft3.xyz ,ft3.xyz, v0 \n" + // multiply by material's diffuse color
// ambiant light
"mov ft4.xyz ,fc0.xyz \n" +
"mul ft4.xyz ,ft4.xyz, fc0.w \n" + // multiply with ambiant light amount
"add ft4.xyz ,ft4.xyz, ft3.xyz \n" + // add with material's diffuse color
//outline
"sge ft5 ,ft7, fc4.z \n"+
"mul ft5.xyz, ft5.xyz, ft4.xyz \n" +
"mov ft5.w ,fc5.w \n" + // alpha value
"mov oc ,ft5 \n"; // output
}
//----------------------------------------------------------------------------------------------
//----------------------------------------------------------------------------------------------
public function setActiveVertextBuffer(context3D:Context3D, vertexBuffer:VertexBuffer3D):void {
// XYZ
context3D.setVertexBufferAt(0, vertexBuffer, 0, Context3DVertexBufferFormat.FLOAT_3);
// colors
context3D.setVertexBufferAt(1, vertexBuffer, 3, Context3DVertexBufferFormat.BYTES_4);
// normales
context3D.setVertexBufferAt(2, vertexBuffer, 4, Context3DVertexBufferFormat.FLOAT_3);
}
//----------------------------------------------------------------------------------------------
//----------------------------------------------------------------------------------------------
public function releaseActiveVertextBuffer(context3D:Context3D):void {
// XYZ
context3D.setVertexBufferAt(0, null);
// colors
context3D.setVertexBufferAt(1, null);
// normales
context3D.setVertexBufferAt(2, null);
}
//----------------------------------------------------------------------------------------------
//----------------------------------------------------------------------------------------------
public function assembleNupload(context3D:Context3D):void {
assemble(context3D);
upload();
}
//----------------------------------------------------------------------------------------------
//----------------------------------------------------------------------------------------------
public function assemble(context3D:Context3D):void {
_vertexShaderAssembler = new AGALMiniAssembler();
_vertexShaderAssembler.assemble(Context3DProgramType.VERTEX, _vertexProgram);
_fragmentShaderAssembler = new AGALMiniAssembler();
_fragmentShaderAssembler.assemble(Context3DProgramType.FRAGMENT, _fragmentProgram);
_shaderProgram = context3D.createProgram();
}
//----------------------------------------------------------------------------------------------
//----------------------------------------------------------------------------------------------
public function upload():void {
_shaderProgram.upload(_vertexShaderAssembler.agalcode, _fragmentShaderAssembler.agalcode);
}
//----------------------------------------------------------------------------------------------
//----------------------------------------------------------------------------------------------
public function destroy():void {
_shaderProgram.dispose();
}
//----------------------------------------------------------------------------------------------
public function get vertexProgram():String{return _vertexProgram;}
public function get fragmentProgram():String { return _fragmentProgram; }
public function get shaderProgram():Program3D {return _shaderProgram;}
public function get dataPerVertex():int {return _dataPerVertex;}
}
}
Any questions, any comments, don’t hesitate.
Moussa Dembélé
Commentaires récents
textures with stage3D – Agal
textures with stage3D – Agal
textures with stage3D – Agal