Followup: Normal Mapping Without Precomputed Tangents

This post is a follow-​up to my 2006 ShaderX5 arti­cle [4] about nor­mal map­ping with­out a pre-​computed tan­gent basis. In the time since then I have refined this tech­nique with lessons learned in real life. For those unfa­mil­iar with the topic, the moti­va­tion was to con­struct the tan­gent frame on the fly in the pixel shader, which iron­i­cally is the exact oppo­site of the moti­va­tion from [2]:

Since it is not 1997 any­more, doing the tan­gent space on-​the-​fly has some poten­tial ben­e­fits, such as reduced com­plex­ity of asset tools, per-​vertex band­width and stor­age, attribute inter­po­la­tors, trans­form work for skinned meshes and last but not least, the pos­si­bil­ity to apply nor­mal maps to any pro­ce­du­rally gen­er­ated tex­ture coor­di­nates or non-​linear deformations.

Inter­mis­sion: Tan­gents vs Cotangents

The way that nor­mal map­ping is tra­di­tion­ally defined is, as I think, flawed, and I would like to point this out with a sim­ple C++ metaphor. Sup­pose we had a class for vec­tors, for exam­ple called Vector3, but we also had a dif­fer­ent class for cov­ec­tors. The lat­ter are vec­tors that are made from the coef­fi­cients of a plane equa­tion, and they trans­form dif­fer­ently than ordi­nary vec­tors. As you may know, nor­mal vec­tors are an exam­ple of cov­ec­tors. Sup­pose the class for cov­ec­tors was called Covector3, hav­ing the same func­tion­al­ity as Vector3, but not com­pat­i­ble for assign­ment. Now imag­ine the fol­low­ing function:

Vector3 tangent;
Vector3 bitangent;
Covector3 normal;
 
Covector3 perturb_normal( float a, float b, float c )
{
    return a * tangent +
           b * bitangent + 
           c * normal; 
           // ^^^^ compile-error: type mismatch for operator +
}

The above func­tion mixes vec­tors and cov­ec­tors in a sin­gle expres­sion, which in this fic­tional exam­ple leads to a type mis­match error. If the normal is of type Covector3, then the tangent and the bitangent should be too, oth­er­wise they can­not form a con­sis­tent frame, can they? In real life shader code of course, every­thing would be defined as float3 and be fine, or rather not.

Math­e­mat­i­cal Com­pile Error

Unfor­tu­nately, the above mis­match is exactly how the ‘tan­gent frame’ for the pur­pose of nor­mal map­ping was intro­duced by the authors of [2]. This type mis­match is invis­i­ble as long as the tan­gent frame is orthog­o­nal. When the exer­cise is how­ever to recon­struct the tan­gent frame in the pixel shader, as this arti­cle is about, then we have to deal with a non-​orthogonal screen pro­jec­tion. This is the rea­son why in the book I had intro­duced both \mathbf{T} (which should be called co-​tangent) and \mathbf{B} (now it gets some­what silly, it should be called co-​bi-​tangent) as cov­ec­tors, oth­er­wise the algo­rithm does not work. I have to admit that I could have been more artic­u­late about this detail. This has caused real con­fu­sion, cf from gamedev.net:

The dis­crep­ancy is explained above, as my ‘tan­gent vec­tors’ are really cov­ec­tors. The def­i­n­i­tion on page 132 is con­sis­tent with that of a cov­ec­tor, and so the frame \left(\mathbf{T}|\mathbf{B}|\mathbf{N}\right) should be called a cotan­gent frame.

Inter­mis­sion 2: Blinns Per­turbed Nor­mals (His­tory Channel)

In this sec­tion I would like to show how the def­i­n­i­tion of \mathbf{T} and \mathbf{B} as cov­ec­tors fol­lows nat­u­rally from Blinns orig­i­nal bump map­ping paper [1]. Blinn con­sid­ers a curved para­met­ric sur­face, for instance, a Bezier-​patch, on which he defines tan­gent vec­tors \mathbf{p}_u and \mathbf{p}_v as the deriv­a­tives of the posi­tion \mathbf{p} with respect to u and v.

In this con­text it is a con­ven­tion to use sub­scripts as a short­hand for par­tial deriv­a­tives, so he is really say­ing \mathbf{p}_u = \partial \mathbf{p} / \partial u, etc. He also intro­duces the sur­face nor­mal \mathbf{N} = \mathbf{p}_u \times \mathbf{p}_v and a bump height func­tion f, which is used to dis­place the sur­face. In the end, he arrives at a for­mula for a first order approx­i­ma­tion of the per­turbed normal:

\[\mathbf{N}' \simeq \mathbf{N} + \frac{f_u \mathbf{N} \times \mathbf{p}_v + f_v \mathbf{p}_u \times \mathbf{N}}{|\mathbf{N}|} ,\]

I would like to draw your atten­tion towards the terms \mathbf{N} \times \mathbf{p}_v and \mathbf{p}_u \times \mathbf{N}. They are the per­pen­dic­u­lars to \mathbf{p}_u and \mathbf{p}_v in the tan­gent plane, and can be seen as the ‘off­set vec­tors’ that ulti­mately dis­place the nor­mal. They are also cov­ec­tors (why, make the duck test: if it trans­forms like a cov­ec­tor, it is a cov­ec­tor) so adding them to the nor­mal does not raise said type mis­match. If we divide these terms one more time by |\mathbf{N}| and flip their signs, we’ll arrive at the ShaderX5 def­i­n­i­tion of \mathbf{T} and \mathbf{B} as follows:

\begin{align*} \mathbf{T} &= -\frac{\mathbf{N} \times \mathbf{p}_v}{|\mathbf{N}|^2} = \nabla u, & \mathbf{B} &= -\frac{\mathbf{p}_u \times \mathbf{N}}{|\mathbf{N}|^2} = \nabla v, \end{align*}

\[\mathbf{N}' \simeq \widehat{\mathbf{N}} - f_u \mathbf{T} - f_v \mathbf{B} ,\]

where the hat (as in \widehat{\mathbf{N}}) denotes the nor­mal­ized nor­mal. \mathbf{T} can be inter­preted as the nor­mal to the plane of con­stant u, and like­wise \mathbf{B} as the nor­mal to the plane of con­stant v. There­fore we have three nor­mal vec­tors, or cov­ec­tors, \mathbf{T}, \mathbf{B} and \mathbf{N}, and they are the a basis of a cotan­gent frame. Equiv­a­lently, \mathbf{T} and \mathbf{B} are the gra­di­ents of u and v, which is the def­i­n­i­tion I had used in the book. The mag­ni­tude of the gra­di­ent there­fore deter­mines the bump strength, a fact that I will dis­cuss later when it comes to scale invariance.

A Lit­tle Unlearning

The mis­take of many authors is to unwit­tingly take \mathbf{T} and \mathbf{B} for \mathbf{p}_u and \mathbf{p}_v, which only works as long as the vec­tors are orthog­o­nal. Let’s unlearn ‘tan­gent’, relearn ‘cotan­gent’, and repeat the his­tor­i­cal devel­op­ment from this per­spec­tive: Peercy et al. [2] pre­com­putes the val­ues f_u and f_v (the change of bump height per change of tex­ture coor­di­nate) and stores them in a tex­ture. They call it ‘nor­mal map’, but is a really some­thing like a ‘slope map’, and they have been rein­vented recently under the name of deriv­a­tive maps. Such a slope map can­not rep­re­sent hor­i­zon­tal nor­mals, as this would need an infi­nite slope to do so. It also needs some ‘bump scale fac­tor’ stored some­where as meta data. Kil­gard [3] intro­duces the mod­ern con­cept of a nor­mal map as an encoded rota­tion oper­a­tor, which does away with the approx­i­ma­tion alto­gether, and instead goes to define the per­turbed nor­mal directly as

\[\mathbf{N}' = a \mathbf{T} + b \mathbf{B} + c \widehat{\mathbf{N}} ,\]

where the coef­fi­cients a, b and c are read from a tex­ture. Most peo­ple would think that a nor­mal map stores nor­mals, but this is only super­fi­cially true. This idea of Kil­gard was, since the unper­turbed nor­mal has coor­di­nates (0,0,1), it is suf­fi­cient to store the last col­umn of the rota­tion matrix that would rotate the unper­turbed nor­mal to its per­turbed posi­tion. So yes, a nor­mal map stores basis vec­tors that cor­re­spond to per­turbed nor­mals, but it really is an encoded rota­tion oper­a­tor. The dif­fi­culty starts to show up when nor­mal maps are blended, since this is then an inter­po­la­tion of rota­tion oper­a­tors, with all the com­plex­ity that goes with it (for an excel­lent review, see the arti­cle about Reori­ented Nor­mal Map­ping [5] here).

Solu­tion of the Cotan­gent Frame

The prob­lem to be solved for our pur­pose is the oppo­site as that of Blinn, the per­turbed nor­mal is known (from the nor­mal map), but the cotan­gent frame is unknown. I’ll give a short revi­sion of how I orig­i­nally solved it. Define the unknown cotan­gents \mathbf{T} = \nabla u and \mathbf{B} = \nabla v as the gra­di­ents of the tex­ture coor­di­nates u and v as func­tions of posi­tion \mathbf{p}, such that

\begin{align*} \mathrm{d} u &= \mathbf{T} \cdot \mathrm{d} \mathbf{p} , & \mathrm{d} v &= \mathbf{B} \cdot \mathrm{d} \mathbf{p} , \end{align*}

where \cdot is the dot prod­uct. The gra­di­ents are con­stant over the sur­face of an inter­po­lated tri­an­gle, so intro­duce the edge dif­fer­ences \Delta u_{1,2}, \Delta v_{1,2} and \Delta \mathbf{p_{1,2}}. The unknown cotan­gents have to sat­isfy the constraints

\begin{align*} \Delta u_1 &= \mathbf{T} \cdot \Delta \mathbf{p_1} , & \Delta v_1 &= \mathbf{B} \cdot \Delta \mathbf{p_1} , \\ \Delta u_2 &= \mathbf{T} \cdot \Delta \mathbf{p_2} , & \Delta v_2 &= \mathbf{B} \cdot \Delta \mathbf{p_2} , \\ 0 &= \mathbf{T} \cdot \Delta \mathbf{p_1} \times \Delta \mathbf{p_2} , & 0 &= \mathbf{B} \cdot \Delta \mathbf{p_1} \times \Delta \mathbf{p_2} , \end{align*}

where \times is the cross prod­uct. The first two rows fol­low from the def­i­n­i­tion, and the last row ensures that \mathbf{T} and \mathbf{B} have no com­po­nent in the direc­tion of the nor­mal. The last row is needed oth­er­wise the prob­lem is under­de­ter­mined. It is straight­for­ward then to express the solu­tion in matrix form. For \mathbf{T},

\[\mathbf{T} = \begin{pmatrix} \Delta \mathbf{p_1} \\ \Delta \mathbf{p_2} \\ \Delta \mathbf{p_1} \times \Delta \mathbf{p_2} \end{pmatrix}^{-1} \begin{pmatrix} \Delta u_1 \\ \Delta u_2 \\ 0 \end{pmatrix} ,\]

and anal­o­gously for \mathbf{B} with \Delta v.

Into the Shader Code

The above result looks daunt­ing, as it calls for a matrix inverse in every pixel in order to com­pute the cotan­gent frame! How­ever, many sym­me­tries can be exploited to make that almost dis­ap­pear. Below is an exam­ple of a func­tion writ­ten in GLSL to cal­cu­late the inverse of a 3×3 matrix. A sim­i­lar func­tion writ­ten in HLSL appeared in the book, and then I tried to opti­mize the hell out of it. For­get this approach as we are not going to need it at all. Just observe how the adju­gate and the deter­mi­nant can be made from cross products:

mat3 inverse3x3( mat3 M )
{
    // The original was written in HLSL, but this is GLSL, 
    // therefore
    // - the array index selects columns, so M_t[0] is the 
    //   first row of M, etc.
    // - the mat3 constructor assembles columns, so 
    //   cross( M_t[1], M_t[2] ) becomes the first column
    //   of the adjugate, etc.
    // - for the determinant, it does not matter whether it is
    //   computed with M or with M_t; but using M_t makes it
    //   easier to follow the derivation in the text
    mat3 M_t = transpose( M ); 
    float det = dot( cross( M_t[0], M_t[1] ), M_t[2] );
    mat3 adjugate = mat3( cross( M_t[1], M_t[2] ),
                          cross( M_t[2], M_t[0] ),
                          cross( M_t[0], M_t[1] ) );
    return adjugate / det;
}

We can sub­sti­tute the rows of the matrix from above into the code, then expand and sim­plify. This pro­ce­dure results in a new expres­sion for \mathbf{T}. The deter­mi­nant becomes \left| \Delta \mathbf{p_1} \times \Delta \mathbf{p_2} \right|^2, and the adju­gate can be writ­ten in terms of two new expres­sions, let’s call them \Delta \mathbf{p_1}_\perp and \Delta \mathbf{p_2}_\perp (with \perp read as ‘perp’), which becomes

\[\mathbf{T} = \frac{1}{\left| \Delta \mathbf{p_1} \times \Delta \mathbf{p_2} \right|^2} \begin{pmatrix} \Delta \mathbf{p_2}_\perp \\ \Delta \mathbf{p_1}_\perp \\ \Delta \mathbf{p_1} \times \Delta \mathbf{p_2} \end{pmatrix}^\mathrm{T} \begin{pmatrix} \Delta u_1 \\ \Delta u_2 \\ 0 \end{pmatrix} ,\]

\begin{align*} \Delta \mathbf{p_2}_\perp &= \Delta \mathbf{p_2} \times \left( \Delta \mathbf{p_1} \times \Delta \mathbf{p_2} \right) , \\ \Delta \mathbf{p_1}_\perp &= \left( \Delta \mathbf{p_1} \times \Delta \mathbf{p_2} \right) \times \Delta \mathbf{p_1} . \end{align*}

As you might guessed it, \Delta \mathbf{p_1}_\perp and \Delta \mathbf{p_2}_\perp are the per­pen­dic­u­lars to the tri­an­gle edges in the tri­an­gle plane. Say Hello! They are, again, cov­ec­tors and form a proper basis for cotan­gent space. To sim­plify things fur­ther, observe:

  • The last row of the matrix is irrel­e­vant since it is mul­ti­plied with zero.
  • The other matrix rows con­tain the per­pen­dic­u­lars (\Delta \mathbf{p_1}_\perp and \Delta \mathbf{p_2}_\perp), which after trans­po­si­tion just mul­ti­ply with the tex­ture edge differences.
  • The per­pen­dic­u­lars can use the inter­po­lated ver­tex nor­mal \mathbf{N} instead of the face nor­mal \Delta \mathbf{p_1} \times \Delta \mathbf{p_2}, which is sim­pler and looks even nicer.
  • The deter­mi­nant (the expres­sion \left| \Delta \mathbf{p_1} \times \Delta \mathbf{p_2} \right|^2) can be han­dled in a spe­cial way, which is explained below in the sec­tion about scale invariance.

Taken together, the opimized code is shown below, which is even sim­pler than the one I had orig­i­nally pub­lished, but still higher quality:

mat3 cotangent_frame( vec3 N, vec3 p, vec2 uv )
{
    // get edge vectors of the pixel triangle
    vec3 dp1 = dFdx( p );
    vec3 dp2 = dFdy( p );
    vec2 duv1 = dFdx( uv );
    vec2 duv2 = dFdy( uv );
 
    // solve the linear system
    vec3 dp2perp = cross( dp2, N );
    vec3 dp1perp = cross( N, dp1 );
    vec3 T = dp2perp * duv1.x + dp1perp * duv2.x;
    vec3 B = dp2perp * duv1.y + dp1perp * duv2.y;
 
    // construct a scale-invariant frame 
    float invmax = inversesqrt( max( dot(T,T), dot(B,B) ) );
    return mat3( T * invmax, B * invmax, N );
}

Scale invari­ance

The deter­mi­nant \left| \Delta \mathbf{p_1} \times \Delta \mathbf{p_2} \right|^2 was left over as a scale fac­tor in the above expres­sion. This has the con­se­quence that the result­ing cotan­gents \mathbf{T} and \mathbf{B} are not scale invari­ant, but will vary inversely with the scale of the geom­e­try. It is the nat­ural con­se­quence of them being gra­di­ents. If the scale of the geomtery increases, and every­thing else is left unchanged, then the change of tex­ture coor­di­nate per unit change of posi­tion gets smaller, which reduces \mathbf{T} = \nabla u = \left( \frac{\partial u}{\partial x}, \frac{\partial u}{\partial y}, \frac{\partial u}{\partial z} \right) and sim­i­larly \mathbf{B} in rela­tion to \mathbf{N}. The effect of all this is a dimin­ished per­tu­ba­tion of the nor­mal when the scale of the geom­e­try is increased, as if a height­field was stretched.

Obvi­ously this behav­ior, while totally log­i­cal and cor­rect, would limit the use­ful­ness of nor­mal maps to be applied on dif­fer­ent scale geom­e­try. My solu­tion was and still is to ignore the deter­mi­nant and just nor­mal­ize \mathbf{T} and \mathbf{B} to whichever of them is largest, as seen in the code. This solu­tion pre­serves the rel­a­tive lengths of \mathbf{T} and \mathbf{B}, so that a skewed or stretched cotan­gent space is sill han­dled cor­rectly, while hav­ing an over­all scale invariance.

Non-​perspective optimization

As the ulti­mate opti­miza­tion, I also con­sid­ered what hap­pens when we can assume \Delta \mathbf{p_1} = \Delta \mathbf{p_2}_\perp and \Delta \mathbf{p_2} = \Delta \mathbf{p_1}_\perp. This means we have a right tri­an­gle and the per­pen­dic­u­lars fall on the tri­an­gle edges. In the pixel shader, this con­di­tion is true when­ever the screen-​projection of the sur­face is with­out per­spec­tive dis­tor­tion. There is a nice fig­ure demon­strat­ing this fact in [4]. This opti­miza­tion saves another two cross prod­ucts, but in my opin­ion, the qual­ity suf­fers heav­ily should there actu­ally be a per­spec­tive distortion.

Putting it together

To make the post com­plete, I’ll show how the cotan­gent frame is actu­ally used to per­turb the inter­po­lated ver­tex nor­mal. The func­tion perturb_normal does just that, using the back­wards view vec­tor for the ver­tex posi­tion (this is ok because only dif­fer­ences mat­ter, and the eye posi­tion goes away in the dif­fer­ence as it is constant).

vec3 perturb_normal( vec3 N, vec3 V, vec2 texcoord )
{
    // assume N, the interpolated vertex normal and 
    // V, the view vector (vertex to eye)
    vec3 map = texture2D( mapBump, texcoord ).xyz;
#ifdef WITH_NORMALMAP_UNSIGNED
    map = map * 255./127. - 128./127.;
#endif
#ifdef WITH_NORMALMAP_2CHANNEL
    map.z = sqrt( 1. - dot( map.xy, map.xy ) );
#endif
#ifdef WITH_NORMALMAP_GREEN_UP
    map.y = -map.y;
#endif
    mat3 TBN = cotangent_frame( N, -V, texcoord );
    return normalize( TBN * map );
}
varying vec3 g_vertexnormal;
varying	vec3 g_viewvector;  // camera pos - vertex pos
varying vec2 g_texcoord;
 
void main()
{
    vec3 N = normalize( g_vertexnormal );
 
#ifdef WITH_NORMALMAP
    N = perturb_normal( N, g_viewvector, g_texcoord );
#endif
 
    // ...
}

The green axis

Both OpenGL and DirectX place the tex­ture coor­di­nate ori­gin at the start of the image pixel data. The tex­ture coor­di­nate (0,0) is in the cor­ner of the pixel where the image data pointer points to. Con­trast this to most 3-​D mod­el­ing pack­ages that place the tex­ture coor­di­nate ori­gin at the lower left cor­ner in the uv-​unwrap view. Unless the image for­mat is bottom-​up, this means the tex­ture coor­di­nate ori­gin is in the cor­ner of the first pixel of the last image row. Quite a dif­fer­ence!
An image search on Google reveals that there is no dom­i­nant con­ven­tion for the green chan­nel in nor­mal maps. Some have green point­ing up and some have green point­ing down. My artists pre­fer green point­ing up for two rea­sons: It’s the for­mat that 3ds Max expects for ren­der­ing, and it sup­pos­edly looks more nat­ural with the ‘green illu­mi­na­tion from above’, so this helps with eye­balling nor­mal maps.

Sign Expan­sion

The sign expan­sion deserves a lit­tle elab­o­ra­tion because I try to use signed tex­ture for­mats when­ever pos­si­ble. With the unsigned for­mat, the value 0.5 can­not be rep­re­sented exactly (it’s between 127 and 128). The signed for­mat does not have this prob­lem, but in exchange, has an ambigu­ous encod­ing for -1 (can be either -127 or -128). If the hard­ware is inca­pable of signed tex­ture for­mats, I want to be able to pass it as an unsigned for­mat and emu­late the exact sign expan­sion in the shader. This is the ori­gin of the seem­ingly odd val­ues in the sign expansion.

In Hind­sight

The orig­i­nal arti­cle in ShaderX5 was writ­ten as a proof-​of-​concept. Although the algo­rithm was tested and worked, it was a lit­tle expen­sive for that time. Fast for­ward to today and the pic­ture has changed. I am now employ­ing this algo­rithm in real-​life projects for great ben­e­fit. I no longer bother with tan­gents as ver­tex attrib­utes and all the asso­ci­ated com­plex­ity. For exam­ple, I don’t care whether the COLLADA exporter of Max or Maya (yes I’m rely­ing on COLLADA these days) out­put usable tan­gents for skinned meshes, nor do I bother to import them, because I don’t need them! For the artists, it doesn’t occur to them that an aspect of the asset pipeline is miss­ing, because It’s all nat­ural: There is a geom­e­try, there are tex­ture coor­di­nates and there is a nor­mal map, and just works.

Take Away

There are no ‘tan­gent frames’ when it comes to nor­mal map­ping. A tan­gent frame which includes the nor­mal is log­i­cally ill-​formed. All there is are cotan­gent frames in dis­guise when the frame is orthog­o­nal. When the frame is not orthog­o­nal, then tan­gent frames will stop work­ing. Use cotan­gent frames instead.


[1] James Blinn, “Sim­u­la­tion of wrin­kled sur­faces”, SIGGRAPH 1978
http://research.microsoft.com/pubs/73939/p286-blinn.pdf

[2] Mark Peercy, John Airey, Brian Cabral, “Effi­cient Bump Map­ping Hard­ware”, SIGGRAPH 1997
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.42.4736

[3] Mark J Kil­gard, “A Prac­ti­cal and Robust Bump-​mapping Tech­nique for Today’s GPUs”, GDC 2000
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.537

[4] Chris­t­ian Schüler, “Nor­mal Map­ping with­out Pre­com­puted Tan­gents”, ShaderX 5, Chap­ter 2.6, pp. 131 – 140

[5] Colin Barré-​Brisebois and Stephen Hill, “Blend­ing in Detail”,
http://blog.selfshadow.com/publications/blending-in-detail/

66 thoughts on “Followup: Normal Mapping Without Precomputed Tangents

  1. Out of inter­est, what impli­ca­tions does this have for tangent-​space cal­cu­la­tions when you *bake* normal-​maps (for exam­ple, inside 3dsmax).
    Wouldn’t you have to also use this method when cre­at­ing the orig­i­nal normal-​map?

    • Hi MoP, absolutely, you’re cor­rect in your assumption.

      If your nor­mal map is the result of sam­pling high poly geom­e­try, then the nor­mal map sam­pler should use the same assump­tions about tan­gent space than the in the engine, if you want high­est fidelity. This is gen­er­ally true, whether you use pre­com­puted tan­gents or not.

      If on the other hand you have a painted nor­mal map, let’s say, a generic rip­ple tex­ture, this just fol­lows what­ever tex­ture map­ping is applied to the mesh. If this map­ping is sheared or stretched, and so vio­lates the square patch assump­tion, the algo­rithm from above is still able to light it correctly.

    • Thanks for the fast reply, Chris­t­ian!
      Absolutely, that makes sense… so I’m won­der­ing if, for “per­fect” results, if peo­ple are bak­ing their nor­mal maps in 3dsmax or Maya or XNor­mal or what­ever, those pro­grams will need to have their tan­gent space cal­cu­la­tions updated/​reworked to match the in-​game cal­cu­la­tion?
      I’m guess­ing that cur­rently the 3dsmax cal­cu­la­tion is not exactly synced with the method you describe here… or am I wrong?

    • You would be sur­prised to know that the method in 3dsMax or any­where else is most likely not synced with any engine, even when using the clas­sic method. There is an entire the­sis devoted to this topic (look for Mikkelsen here). If you want such per­fec­tion, then the engine should have its own tools for bak­ing nor­mal maps because only then it is going to know what their shaders do. 3dsMax can never know this!

    • Yep, that makes sense - in fact that’s what the Doom 3 engine (id Tech 4) did to ensure the nor­mal maps were per­fect … they imported LWOs into the engine and baked the normal-​maps using the game engine so that the cal­cu­la­tions would match up per­fectly.
      I haven’t seen any other engine (released pub­li­cally) do that yet, but I work in game devel­op­ment and I know that we have been try­ing to solve this prob­lem for years with no sat­is­fac­tory results. Cur­rently the best method seems to be to mod­ify a third-​party pro­gram (eg. XNor­mal) to cal­cu­late tan­gent space in exactly the same way as the tar­get engine cal­cu­lates it, as this is the only way to ensure the normal-​maps will be absolutely correct.

  2. Pingback: Normal Mapping Without Precomputed Tangents « Interplay of Light

  3. I am con­fused to what space your TBN matrix the nor­mal from the nor­mal map tex­ture con­verts! You need this in order to do light­ing cal­cu­la­tions. Should I trans­form the light vec­tor and view vec­tor with the TBN matrix, or leave them in cam­era space?

    Do you assume in the frag­ment shader that g_​vertexnormal, g_​viewvector are in the ver­tex shader mul­ti­plied with the gl_​NormalMatrix and gl_​ModelViewMatrix respec­tively? Oth­er­wise how do you account for the trans­for­ma­tions on the object.

    Thanks

    • Sorry I meant in the pre­vi­ous post “light vec­tor” and “eye vec­tor” instead of view vec­tor for the fist paragraph.

    • Hi cinepi­vates,
      the TBN matrix that I build in the pixel shader is the trans­form from cotan­gent space to world space. After the vec­tor from the nor­mal map is mul­ti­plied with the TBN matrix, it is a world space nor­mal vec­tor. This is due to the fact that both the inter­po­lated ver­tex nor­mal and the view vec­tor are sup­plied in world space by the ver­tex shader (not shown). Alter­na­tively, if the ver­tex shader sup­plies these vec­tors in eye space, the pixel shader should con­struct the TBN matrix to con­vert into eye space instead. So you can choose your way.

    • Hi
      thanks for the reply.
      Since you are mul­ti­ply­ing per pixel with a mat3 (TBNxNor­mal) on the pixel shader have you mea­sured any per­for­mance decrease com­pared to the more tra­di­tional method of pre­com­puted Tan­gents where you usu­ally only trans­form in the ver­tex shader the light,eye vec­tors into tan­gent space using the TBN matrix.

    • I don’t use the TBN in the ver­tex shader. That is a thing of the past when there was pixel shader model 1.x. It pre­vents you to use world space con­stants in the pixel shader. For exam­ple, a reflec­tion vec­tor to look up a world space envi­ron­ment map. So the matrix mul­ti­ply by TBN in the pixel shader has always been there for me.

  4. Hi,I just won­der­ing if light­ing cal­cu­la­tions can be done in tan­gent space.My tex­tures could not match the Square Patch Assumption,but I don’t want to lose tan­gent space lighting,since some tech­niques are par­tic­u­lar designed for that.

    • Hi April,
      this is tricky, as light­ing in tan­gent space does only work if it is orthog­o­nal. Think about it: the angle between any vec­tors (say, the angle between \mathbf{N} and \mathbf{L}) will change under a non-​orthogonal trans­for­ma­tion. So dot(N,L) is going to result in a dif­fer­ent value, depend­ing on which space it is in. If you can live with that, then take the inverse-​transpose of the TBN matrix (this is in essence what you get when you do the non-​perspective opti­miza­tion men­tioned in the post), and use that to trans­form \mathbf{L}, \mathbf{V}, \mathbf{H} and any other vec­tor you need into tan­gent space, and do the light­ing com­pu­ta­tions there.

    • Thanks for reply :)
      I man­aged to orthog­o­nal­ize the matrix and then used the inverse-​transpose one to do light­ing in world space, the result became a mess.I checked the orgin matrix, and found out the matrix can not be inversed where UV mir­rors. The deter­mi­nant() func­tion returns 0.
      Still look­ing for reason.

    • I’m not sure I can fol­low your argu­ment. If you want to do light­ing in world space, then you don’t need to change any­thing. If you want to do light­ing in tan­gent space instead (which only yields sim­i­lar results if the tan­gent space is roughly orthog­o­nal), you’d need a matrix to con­vert your light, view, etc vec­tors into tan­gent space. Edit: For this you need the inverse of the TBN. You can get the inverse-​tranpose already very sim­ply by ignor­ing the two cross prod­ucts and using dp2perp = dp1 and dp1perp = dp2. Then the trans­pose of this would be the inverse of the TBN (ignor­ing scale). In the shader you don’t need to explic­itly trans­pose, you can just mul­ti­ply with the vec­tors from the left (eg vector * matrix instead of transpose(matrix) * vector). Then you can orthog­o­nal­ize this matrix if you want, but this won’t help much if the tan­gent space is not orthog­o­nal to begin with. I don’t under­stand why you are will­ing to go though such hoops instead of sim­ply doing the light­ing in world space.

  5. Fan­tas­tic arti­cle! I switched my engine from using pre­cal­cu­lated tan­gents to the method your describe - very easy to imple­ment. I’m using WebGL, but I’m pre­sum­ing my find­ings will cor­re­spond to what OpenGL + GLES pro­gram­mers see. On desk­top, I don’t really notice much of a per­for­mance dif­fer­ence (if any­thing, the tangent-​less approach is slightly slower). But on mobile (tried on both Galaxy Nexus and Nexus 7), this method roughly twice as slow as using pre­com­puted tan­gents. Seems dFdx and dFdy are par­tic­u­larly slow on mobile GPUs. Just cut­ting out those calls takes my FPS from ~24FPS to ~30FPS. So although I think this method is ele­gant to the extreme, I’m not sure it’s fast enough to be the bet­ter option (at least, not on mobile). I really hope some­one can con­vince me oth­er­wise though! :)

    • Hi Will, thanks for shar­ing. Of course those addi­tional ~14 shader instruc­tions for cotangent_frame() are not free, espe­cially not on mobile (which is like 2005 desk­top, the time when the arti­cle was orig­i­nally writ­ten). For me today, this cost is invis­i­ble com­pared to all the other things that are going on, like mul­ti­ple lights and shad­ows and so forth. On the newest archi­tec­tures like the NVidia Fermi, it could already be a per­for­mance win to go with­out pre­com­puted tan­gents, due to the rea­sons men­tioned in the intro­duc­tion. But while that is nice, the main rea­son I use the method is the boost in productivity.

  6. Pingback: Martin Codes – Cool Link Stash, January 2013

  7. For me the equa­tions in this are screwed up. The are ran­dom LaTeX things in the images (Delta­mathbf). Same with Safari, Chrome and Firefox.

    Would be nice if you could fix that.

  8. Hallo there, I have fixed the math for­mu­lae in the post. It turned out to be an incom­pat­i­bil­ity between two plu­g­ins, and that got all \LaTeX back­slashes eaten. Sorry for that! Should there be a bro­ken for­mula that I have over­looked, just drop a line.

  9. So glad I found your blog. Though I am an artist these more tech­ni­cal insights really help me in under­stand­ing what to com­mu­ni­cate to our coders to estab­lish cer­tain looks.

  10. Hi, I want to ask you to elab­o­rate on 2 fol­low­ing moments. First of all I would like to know why cross(N,Pv)/length(N)=gradient(u) and cross(Pu, N)/length(N)=gradient(v). I just can not find any math­e­mat­i­cal deriva­tion of this fact. Also it is unclear for me why deltaU = dot(T, deltaP), and how this fol­lows from def­i­n­i­tion of gra­di­ent. Oth­er­wise I found this arti­cle very excit­ing and my exper­i­ments showed that imple­men­ta­tion works very well and is a good drop-​in replace­ment for con­ven­tional tan­gent basis.

    • Hi Mykhailo,
      thanks for shar­ing your expe­ri­ence. Indeed, I did not explain why \mathbf{N} \times \mathbf{P}_v / |\mathbf{N}|^2 = \nabla u, and that may not be obvi­ous, so here it goes: The gra­di­ent vec­tor is always per­pen­dic­u­lar to the iso-​surface (aka. “level-​set”). In a skewed 2-​D coor­di­nate sys­tem, the iso-​line of one coor­di­nate is sim­ply the other coor­di­nate axis! So the gra­di­ent vec­tor for the u tex­ture coor­di­nate must be per­pen­dic­u­lar to the v axis, and vice versa. (The scale fac­tor makes it such that the over­all length equals the rate of change, which is depen­dent on the assump­tion that |\mathbf{N}| = |\mathbf{P}_u \times \mathbf{P}_v|.) Your sec­ond ques­tion is also related to the fact that the gra­di­ent vec­tor is per­pen­dic­u­lar to the iso-​surface. If a posi­tion delta is made par­al­lel to the iso-​surface, then the tex­ture coor­di­nate doesn’t change, because in this case the dot prod­uct is zero.

    • Thank you for an answer, now every­thing is a bit clearer. Still was able to fully fig­ure it out only after I under­stood that para­metri­sa­tion of u, v is lin­ear in the plane of every tri­an­gle. For some rea­son I just missed that fact. And after you men­tioned that gra­di­ent is per­pen­dic­u­lar to iso-​line every­thing made sense. And now it make sense why change of u is pro­jec­tion of posi­tion delta onto gra­di­ent vector.

  11. Hi Chris­t­ian, thanks for sharing!

    I am work­ing on an enhanced ver­sion of the Crytek-​Sponza scene and I had prob­lems with my per-​pixel nor­malmap­ping shader based on pre­com­puted tan­gents. I had most sur­faces lighted cor­rectly, but in some cases they weren’t. It turned out that - unknown to me - some faces had flipped UVs (which is not that uncom­mon) and there­fore the cotan­gent frame was messed up, because the inverse of the tan­gent was used to cal­cu­late the binor­mal as the crossprod­uct with the ver­tex normal.

    Now, I replaced it with your approach and every­thing is alright; it works instantly and all errors van­ished. Awesome!! :-)

  12. Hi,

    Does the com­pu­ta­tion still work if the mesh nor­mal is in view space, and the g_​viewvector = vec3(0, 0, -1) ?

    Cheers :)

    • Yes, but you must pro­vide the ver­tex posi­tion in view space also. The view vec­tor is used as a proxy to dif­fer­en­ti­ate the ver­tex posi­tion, there­fore a con­stant view vec­tor will not do.

    • Ah ok, I just nor­mal­ized the view posi­tion and its work­ing perfectly

      Thanks!

  13. Hi,
    Your idea seem to be very inter­est­ing.
    But I’ve tried it in real scene, and found a glitch - if uv (tex­ture coor­di­nates) are mir­rored, nor­mal is also become mirrored.

    Is there any way to fix it?

    • The nor­mal itself should not be flipped, only the tan­gents. If the u tex­ture coor­di­nate is mir­rored, then the \mathbf{T} tan­gent should reverse sign and sim­i­lar with v and the \mathbf{B} tangent.

  14. In the code you show in the arti­cle, you’re using -V as the posi­tion for tak­ing deriv­a­tives. But V is nor­mal­ized per-​pixel, so your posi­tion deriv­a­tives effec­tively have their radial com­po­nent (toward/​away from the cam­era) pro­jected out. Doesn’t this cre­ate some arti­facts when the sur­face is at a glanc­ing angle to the camera?

    (You also lose scale infor­ma­tion, but since you’re nor­mal­iz­ing to make it scale-​invariant any­way, I sup­pose this does not matter.)

    • Hi Nathan the arti­cle doesn’t men­tion it explic­itly but the view vec­tor is meant to be passed unnor­mal­ized from ver­tex to pix­elshader. The nor­mal­iza­tion should hap­pen after it has been used to com­pute the cotan­gent frame.
      (EDIT: I cor­rected the nor­mal­iza­tion mis­take in the exam­ple code, now code + text are in agreement)

  15. Hello Chris­t­ian,

    What can be done with the tex­co­ords if we sam­ple from 2 nor­mal maps with dif­fer­ent uv’s

    We add them? or? i guess the rate of change will be the same

    Exam­ple code is:
    float3 normal1 = tex2d(normalmap1, 3*uv1);
    float3 normal2 = tex2d(normalmap2, 5*uv2);

    float3 nor­mal = nor­mal­ize (lerp(normal1, normal2, 0.5f) );

    Remarks: This oper­a­tion is not good blend­ing at all and my ques­tion is, if it was, how i will can use the uv’s?

    Best Regards

    • float2 uv_​for_​perturbation_​function = lerp(3*uv1, 5*uv2, 0.5f)

      Like this?

    • Hi Ste­fan,
      I assume that the map­ping of uv1 and uv2 is dif­fer­ent. Then you are going to need a tan­gent frame for each, trans­form both nor­mals into a com­mon space (here: world space) and then mix them.


      float3 normalmap1 = tex2d(normalmap1, 3*uv1);
      float3 normalmap2 = tex2d(normalmap2, 5*uv2);

      float3x3 TBN1 = cotangent_​frame( vertex_​normal, -viewvec­tor, uv1 );
      float3x3 TBN2 = cotangent_​frame( vertex_​normal, -viewvec­tor, uv2 );

      float3 nor­mal = nor­mal­ize( lerp( mul( TBN1, normalmap1 ), mul( TBN2, normalmap2 ), 0.5f ) );

      etc
      hope this helps

  16. Hello,

    I try to do impor­tance sam­pling of the Beck­mann Dis­tri­b­u­tion of a Cook Tor­rance BRDF. I use your descrip­tion to cre­ate a TBN-​Matrix in order to trans­form the halfvec­tor from tangent-​space to world-​space.
    The prob­lem is that I can see every frag­ment (tri­an­gle) of my geom­e­try and not a glossy shaded sur­face. I’m not sure if the prob­lem is really con­nected with the TBN-​Matrix. Does it work for light­ing cal­cu­la­tions with­out restric­tions? Or do you think I have to search the mis­take some­where else?

    • Hallo Phil,
      the par­tial deriv­a­tives of tex­ture coor­di­nates (the dFdx and dFdy instruc­tions) are con­stant over the sur­face of a tri­an­gle, so both tan­gent direc­tions, \mathbf{T} and \mathbf{B}, are going to be faceted (aka ‘flat shaded’). How­ever the nor­mal vec­tor \mathbf{N} is not. So unless you use are very low spec­u­lar expo­nent (high RMS slope in case of Beck­mann) it should not visu­ally matter.

  17. I tried this, but am hav­ing some prob­lems.
    I’m using a left-​handed coor­di­nate sys­tem and I pro­vide the nor­mal and the view vec­tor in view space. It seems from one of the com­ments, that this should work.
    At first I though every is look­ing good, but than I noticed, that the per­turbed nor­mals are not quite right in some cases. Light­ing code should be fine, because results with unper­turbed nor­mals look as expected.
    If I use the cotan­gent frame to trans­form the view vec­tor into cotan­gent space for par­al­lax map­ping, the results look cor­rect as well.
    I’m also using D3D-​style UVs where (0,0) is the top left cor­ner, but that should not mat­ter, right?
    Any ideas what could wrong?

    • Hi Ben­jamin,
      as I said in the arti­cle, the tex­ture coor­di­nate ori­gin is at the start of the image array in both OpenGL and D3D so the UV mir­ror­ing must be done in both APIs. If your coor­di­nate sys­tem is left handed, you’ll need to negate both \mathbf{T} and \mathbf{B}, that’s all.

  18. I’ve tried imple­ment­ing this, but found that you get a faceted appear­ance on the world space nor­mal, pre­sum­ably due to the deriv­a­tives being per-​triangle. Am I doing some­thing wrong, or is this a lim­i­ta­tion of this technique?

    • Hi James,
      as some other peo­ple have com­mented, the tan­gen­tial direc­tions are faceted, due to fact tha par­tial deriv­a­tives (dFdx) of tex­ture coor­di­nates are con­stant per tri­an­gle. How­ever for the nor­mal direc­tion the inter­po­lated nor­mal is used so the faceting can only appears if there is a dif­fer­ence in the UV gra­di­ents from one tri­an­gle to the next.

  19. Could you please post your ver­tex shader code? I’m doing some­thing dumb wrong and I’ve been try­ing to get this to work all day.

    • Hi Rob,
      there is really noth­ing to the ver­tex shader, just trans­forms. Below is an exam­ple (transforms[0] is the model-​to-​world trans­form and transforms[1] is world-​to-​clip).


      uniform mat4 transforms[2];
      uniform vec4 camerapos;
      varying vec2 texcoord;
      varying vec3 vertexnormal;
      varying vec4 viewvector;

      void main()
      {
      vec4 P = transforms[0] * gl_​Vertex;
      gl_​Position = transforms[1] * P;
      tex­co­ord = gl_MultiTexCoord0.xy;
      ver­texnor­mal =
      ( transforms[0] * vec4( gl_​Normal, 0. ) ).xyz;
      viewvec­tor = cam­er­a­pos - P;
      }

  20. Ohhh, I was hop­ing my prob­lem might be in the ver­tex shader code but that’s what I have.

    The prob­lem I have is that if I use the nor­mal from perturb_​normal, the light­ing rotates with the model so it is always the same side of the model that is lit (beau­ti­fully) no mat­ter how the model is ori­ented rel­a­tive to the light source. If I light the model with just the inter­po­lated ver­tex nor­mal I get light­ing which works as expected.

    I wrote out the details of what I’m doing at http://onemanmmo.com/index.php?cmd=newsitem&comment=news.1.158.0 I’m going to have to look at this again tomor­row to see if I can fig­ure out what it is I’m doing wrong. Thank you for your help.

  21. (This is a follow-​up to my pre­vi­ous post, but I can’t find out how to reply to your reply.)

    I noticed you say­ing
    “Hi Nathan the arti­cle doesn’t men­tion it explic­itly but the view vec­tor is meant to be passed unnor­mal­ized from ver­tex to pix­elshader. The nor­mal­iza­tion should hap­pen after it has been used to com­pute the cotan­gent frame.” In one of the replies.
    In the code how­ever the view vec­tor is nor­mal­ized before it is passed to per­turb­Nor­mal(). Isn’t this contradictory?

    The rea­son I’m ask­ing is that I’m still strug­gling to get this work­ing for view space nor­mals. If the nor­malmap only con­tains (0, 0, 1), the final per­turbed nor­mal is unchanged and every­thing works as expected, as the nor­mal effec­tively is not per­turbed at all.
    Per­turb­ing by (0, 1, 0) how­ever gives me strange results for instance. It is espe­cially notice­able at graz­ing angles. Look­ing at a plane there is a ver­ti­cal line where the nor­mal sud­denly flips dras­ti­cally although the sur­face nor­mal did not change at all.
    I tried flip­ping B and T etc., but the “flip­ping” effect is not imme­di­ately related to this.

    • Hi Ben­jamin,
      thanks for the sug­ges­tion, that’s a seri­ous gotcha, and I cor­rected it.
      The code that I posted should pass g_viewvector into the func­tion perturb_normal.
      If you fol­low the argu­ment in the arti­cle it should be obvi­ous: perturb_normal offi­cially wants the ver­tex posi­tion, but any con­stant off­set to that is going to can­cel when tak­ing the deriv­a­tive. There­fore, the view vec­tor can be passed in as a sur­ro­gate of the ver­tex posi­tion, but this is only true of the unnor­mal­ized view vector.

      Have you tried that?

      (BTW, while you can­not reply to my reply, you can reply to your orig­i­nal com­ment. The max­i­mum nest­ing level here is 2)

  22. Hi,
    I’m new to shaders, but I have to imple­ment some fea­tures in a webGL project (based on this tem­plate: http://learningwebgl.com/lessons/lesson14/index.html)
    I have already imple­mented a spec­u­lar map, but here are some new things and another shader language.

    What I actu­ally want to ask:
    - where comes the ‘map­Bump’ from? Is it already imple­mented in GLSL (and webGL)?
    - How does ‘map = map * 255./127. - 128./127.;’ work? What means the ‘.’ after each number?

    • Nev­er­mind!
      In the end, every­thing was self-​explaning.
      Also I just got it work­ing.
      The webGL shader lan­guage seems to be a bit imma­ture which caused many issues, but finally, I got it working.

      Thanks for the great tutorial!

    • Hallo Basti,
      WebGL is mod­eled after OpenGL ES (and not Desk­top OpenGL), so there may be some incom­pat­i­bil­i­ties. But the shad­ing lan­guage as such should be the same GLSL. Of course, the name ‘map­Bump’ is only an exam­ple and you can use any old name. You should be able to leave out a lead­ing or trail­ing zero in float­ing point num­bers, so you can write 2. instead of 2.0 etc.

  23. Hi Chris­t­ian,
    Thank you for writ­ing this arti­cle to explain the con­cept of the cotan­gent frame, and also for your atten­tion to imple­men­ta­tion effi­ciency details.

    I’m still hazy on a num­ber of things though, espe­cially these two:
    1.) In the Inter­mis­sion 2 sec­tion, you write:
    T = cross(N, Pv)/length(N)^2 = gradient(u)
    B = cross(Pu, N)/length(N)^2 = gradient(v)
    You seem to imply that if the tan­gent frame is orthog­o­nal, T = Pu and B = Pv, and the dif­fer­ences only occur when the frame has skew. How­ever, I’m con­fused about the con­ven­tions you’re using: If we assume cross prod­ucts fol­low the right-​hand rule, and if we assume Pu points right and Pv points up, then T = -Pu and B = -Pv for an orthog­o­nal tan­gent frame. In other words, T = Pu and B = Pv only holds for orthog­o­nal tan­gent frames if either:
    a.) We’re using the left-​hand rule (thank you Direct3D for ruin­ing con­sis­tent lin­ear alge­bra con­ven­tions for­ever)
    b.) Pv points down (i.e. the tex­ture ori­gin is in the top-​left cor­ner, rather than the bot­tom right cor­ner)
    …but not both.
    What convention/​assumption are you using for the above rela­tion­ships? Are you using the same conventions/​assumptions through­out the rest of the math­e­mat­i­cal deriva­tion? Would it be more appro­pri­ate in a ped­a­gog­i­cal sense to swap the order of the cross prod­ucts, or am I miss­ing some­thing important?

    2.) I don’t have the back­ground to deeply under­stand the pre­cise dif­fer­ence between a tan­gent frame and a cotan­gent frame. Here is the extent of my back­ground knowl­edge on the sub­ject:
    a.) Sur­face nor­mals are covectors/​pseudovectors, what­ever the heck that means.
    b.) Covectors/​pseudovectors like nor­mals trans­form dif­fer­ently from ordi­nary vec­tors. Con­sider matrix M, col­umn vec­tor v, and nor­mal [col­umn] vec­tor n in the same coor­di­nate frame as v. If you use mul(M, v) to trans­form v into another coor­di­nate frame, you need to use mul(transpose(inverse(M)), n) to trans­form the nor­mal vec­tor. This reduces to mul(M, n) if M is orthog­o­nal, but oth­er­wise the inverse-​transpose oper­a­tion is nec­es­sary to main­tain the nor­mal vector’s prop­er­ties in the des­ti­na­tion frame (specif­i­cally, to keep the nor­mal vec­tor per­pen­dic­u­lar to the sur­face). I’ve taken this for granted for some time, but I don’t have a deep under­stand­ing of why the inverse-​transpose is the magic solu­tion for trans­form­ing nor­mal vec­tors, and I believe this might be why I’m hav­ing trou­ble under­stand­ing the nature of the cotan­gent frame.

    Now, if I were to cre­ate a “tra­di­tional” TBN matrix from deriv­a­tives, I’d do it roughly like this (no opti­miza­tion, for clarity…and for­give me if some Cg-​like con­ven­tions slip through):
    vec3 dp1 = dFdx( p );
    vec3 dp2 = dFdy( p );
    vec2 duv1 = dFdx( uv );
    vec2 duv2 = dFdy( uv );
    /​/​set up a lin­ear sys­tem to solve for the TBN matrix:
    vec3 p_​mat = mat3(dp1, dp2, N);
    vec3 uv_​mat = mat3(duv1, duv2, vec3(0.0, 0.0, 1.0));
    /​/​TBN trans­forms [reg­u­lar, non-​normal] vec­tors from
    /​/​tan­gent space to world­space, so:
    /​/​p_​mat = mul(TBN, uv_​mat)
    /​/​there­fore, solve as:
    mat3 TBN = mul(p_mat, inverse(uv_mat));
    /​/​exam­ple trans­forms:
    vec3 arbitrary_​worldspace_​vector = mul(TBN, arbitrary_​tangent_​space_​vector);
    vec3 worldspace_​normal = mul(transpose(inverse(TBN)), normal_​map_​val);

    First, a quick ques­tion: Due to the inverse-​transpose oper­a­tion, this bla­tantly inef­fi­cient “tra­di­tional TBN” solu­tion should work cor­rectly even for non-​orthogonal TBN matri­ces, correct?

    Any­way, it fol­lows from the nature of the tangent-​to-​worldspace trans­for­ma­tion that p_​mat = mul(TBN, uv_​mat) above. After all, the whole point of the world­space edges is that they’re the uv-​space edges trans­formed from tangent-​space to worldspace.

    For this rea­son, I still share some of the con­fu­sion of the poster at http://www.gamedev.net/topic/608004-computing-tangent-frame-in-pixel-shader/. You men­tioned the dis­crep­ancy has to do with your TBN matrix being a cotan­gent frame com­posed of cov­ec­tors, but I’m not yet clear on the impli­ca­tions of that.

    How­ever, I noticed some­thing inter­est­ing that I’m hop­ing will make every­thing fall into place: You trans­form your nor­mal vec­tor *directly* from tangent-​space to world­space using your cotan­gent frame. That is, you just do some­thing like:
    vec3 worldspace_​normal = mul(cotangent_TBN, normal_​map_​val);

    Is the your cotan­gent frame TBN matrix sim­ply the inverse-​transpose of the tra­di­tional tan­gent frame TBN matrix? If so, and cor­rect me if I’m wrong here…if you wanted to trans­form ordi­nary vec­tors like L and V from world­space to tangent-​space (for e.g. par­al­lax occlu­sion map­ping), you would sim­ply do:
    tangent_​space_​V = mul(transpose(cotangent_TBN), worldspace_​V);

    If I’m under­stand­ing all this cor­rectly, con­struct­ing your cotan­gent frame is inher­ently more effi­cient than the tra­di­tional TBN for­mu­la­tion for non-​orthogonal matri­ces: It lets us trans­form [co]tangent-space nor­mals to world­space nor­mals with a sim­ple TBN mul­ti­pli­ca­tion (instead of an inverse-​transpose TBN mul­ti­pli­ca­tion), and it lets us trans­form ordi­nary vec­tors from world­space t [co]tangent-space with a transpose-​TBN mul­ti­pli­ca­tion instead of an inverse-​TBN multiplication.

    Do I have this cor­rect, or is the cotan­gent frame matrix some­thing entirely dif­fer­ent from what I am now thinking?

    • Hi Michael,
      that’s a hand full of a com­ment so i’m try­ing my best to answer.

      (1)
      Good obser­va­tion! The for­mula which says \mathbf{T} = \mathbf{N} \times \mathbf{P_v} has the cross prod­uct back­wards, and I should cor­rect it in the arti­cle. Note that the code in the shader does the right thing, e.g. it says dp2perp = cross( dp2, N ), since my deriva­tion is based on the assump­tion that \mathbf{T} behaves the way I describe it.

      So since I took the for­mula out of Blinns Paper, I checked it to see what he has to says about his cross prod­uct busi­ness — and in one of his draw­ings, the vec­tor \mathbf{N} \times \mathbf{P_u} is indeed in the oppo­site direc­tion as \mathbf{P_v}:

      (2)
      Co-​vectors are a fancy name for “coor­di­nate func­tions”, i.e. plane equa­tions (as in a x + b y + c z + \dots). The a, b, c, \dots behave like any old vec­tor space and so they’re called the coor­di­nates of a co-​vector, while the x, y, z, \dots are the coor­di­nates of an ordi­nary vector.

      The ordi­nary vec­tor tells you “the x axis is in this direc­tion”. The co-​vector tells you the func­tion for the x-​coordinate.

      If you have a matrix of col­umn basis vec­tors, then the rows of the inverse matrix are the co-​vectors, the plane equa­tions that give the coor­di­nate functions.

      If you stick with the dis­tinc­tion to treat co-​vectors as rows, you don’t need any of this “inverse trans­pose” busi­ness - just the ordi­nary inverse will do, and you mul­ti­ply nor­mal vec­tors as rows from the left.

      (2b)
      If your TBN matrix was con­structed as (\mathbf{P_u} | \mathbf{P_v} | \mathbf{N}), then yes, you would need to take the inverse (trans­pose) of that to cor­rectly trans­form a nor­mal vec­tor in the gen­eral case. But the whole point of the arti­cle is to con­struct the TBN matrix as (\mathbf{T} | \mathbf{B} | \mathbf{N}) instead, which you can use directly, since it is a co-​tangent frame!

      The author of the post at http://www.gamedev.net/topic/608004-computing-tangent-frame-in-pixel-shader/ uses the aster­isk sym­bol “*” to describe a matrix prod­uct in the first case, and a dot prod­uct in the sec­ond case. He won­ders why, seem­ingly, the for­mula to com­pute the same thing has the “prod­uct” on dif­fer­ent sides by dif­fer­ent authors. That’s his “dis­crep­ancy”. The truth is: The for­mu­lae do not com­pute the same thing!

      Is the your cotan­gent frame TBN matrix sim­ply the inverse-​transpose of the tra­di­tional tan­gent frame TBN matrix?

      Yes, you can say that \mathbf{T} and \mathbf{B} are the “inverse-​transpose” of \mathbf{P_u} and \mathbf{P_v}. The third one, \mathbf{N}, does not change direc­tion under the inverse-​transpose oper­a­tion, since it is orthog­o­nal to the other two. And if it is unit length, it will stay unit length.

  24. Excel­lent! Thank you so much for tak­ing the time to answer my ques­tions. I had started to sus­pect that the inverse-​transpose of the Pu|Pv|N matrix was a round­about way of com­put­ing what you com­pute directly, and I’m very glad to hear that’s the case. I still have some learn­ing to do to com­fort­ably manip­u­late cov­ec­tors, but it helps a great deal that the end result is some­thing I recognize…and under­stand­ing cov­ec­tors more thor­oughly will finally take the “black magic” out of why the inverse-​transpose of Pu|Pv|N also works (much less effi­ciently of course).

  25. Pingback: Spellforce 2 Demons of the Past | The Tenth Planet

  26. Pingback: Decals (deferred rendering) | IceFall Games

  27. Pingback: Balancing | Spellcaster Studios

  28. Hi there! I’m hav­ing trou­ble using the func­tions dFdx() and dFdy()… tried adding this line:
    #exten­sion GL_​OES_​standard_​derivatives : enable
    in my shader, but it gives me the mes­sage that the exten­sion isn’t sup­ported. Do you have any idea about how i’m sup­posed to do this?

  29. Hi Chris­t­ian,
    I’m research­ing on the run-​time gen­er­ate tbn matrix related top­ics recently, and I found your arti­cle very inter­est­ing. I’m won­der­ing if I want to inte­grate your glsl shader with my code, which is writ­ten in directx/​hlsl, should I use the -N instead in the final result of cotan­gent frame matrix?(for deal­ing with the Right-​handed and left-​handed issue). If this is not the solu­tion, what should I do? Thanks.

    • Hi Sherry
      the one gotcha you need to be aware of is that HLSL’s ddy has dif­fer­ent sign than dFdy, due to OpenGL’s win­dow coor­di­nates being bottom-​to-​top (when not ren­der­ing into an FBO, that is). Other than that, it is just syn­tac­ti­cal code con­ver­sion. For test­ing, you can sub­sti­tute N with the face nor­mal gen­er­ated by cross­ing dp1 and dp2; that must work in every case, what­ever sign con­ven­tion the screen space deriv­a­tives have.

  30. Hi Chris­t­ian!
    First, thanks for an explana­tory arti­cle, it’s great!
    I still have a cou­ple of ques­tions. Lets list some facts:
    a) You said in response to Michael that transpose(TBN) can be used to trans­form eg. V vec­tor from world-​space to tangent-​space.
    b) dFdx and dFdy, and chence dp1/​2 and duv1/​2 are con­stant over a tri­an­gle.
    Based on that facts, can your TBN be com­puted in geometry-​shader on per-​triangle basis, and its trans­pose used to trans­form V, L, Blinn’s half vec­tor, etc. to tangent-​space, in order to make “clas­sic” light­ning in pixel-​shader?
    I’ve found some­thing sim­i­lar in: http://www.slideshare.net/Mark_Kilgard/geometryshaderbasedbumpmappingsetup
    but there is noth­ing about com­mon tangent-​basis cal­cu­la­tion in tex­ture bak­ing tool and shader. The lat­ter is necce­sary, since sub­sti­tut­ing nor­mal map with flat normal-​map (to get sim­ple light­ning) pro­duces faceted look, as in Kilgard’s approach. The sec­ond ques­tion: do you have a plu­gin for xNor­mal, which can com­pute cor­rect per-​triangle tan­gent basis?

    • Hi Andrzej,
      yes, you can (and I have) do the same cal­cu­la­tions in the geom­e­try shader. The cotan­gent basis is always com­puted from a tri­an­gle. In the pixel shader, the ‘tri­an­gle’ is implic­itly spanned between the cur­rent pixel and neigh­bor­ing pix­els. In the geom­e­try shader, you use the actual tri­an­gle, i.e., dFdx and dFdy are sub­sti­tuted with the actual edge dif­fer­ences. The rest will be iden­ti­cal.
      You can then pass TBN down to the pixel shader, or use it to trans­form other quan­ti­ties and pass these.

      Faceting: As I said before, the T and B vec­tors will be faceted, but the N vec­tor can use the inter­po­lated nor­mal to give a smoother look. In prac­tice, if the UV map­ping does not stray too far from the square patch assump­tion, the faceting will be unnoticeable.

  31. Thank You very much, Christian!

    I imple­mented the tech­nique and wrote XNor­mal plu­gin, yet, when trans­form­ing gen­er­ated tangent-​space nor­mal map back to object-​space (by XNor­mal tool), I didn’t get the expected result. I’ll try to make per-​triangle tbn com­pu­ta­tion order-​independent and check the code.

Leave a Reply

Your email address will not be published.


× 8 = fifty six

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>