Followup: Normal Mapping Without Precomputed Tangents

This post is a fol­low-up to my 2006 ShaderX5 arti­cle [4] about nor­mal map­ping with­out a pre-com­put­ed tan­gent basis. In the time since then I have refined this tech­nique with lessons learned in real life. For those unfa­mil­iar with the top­ic, the moti­va­tion was to con­struct the tan­gent frame on the fly in the pix­el shad­er, which iron­i­cal­ly is the exact oppo­site of the moti­va­tion from [2]:

Since it is not 1997 any­more, doing the tan­gent space on-the-fly has some poten­tial ben­e­fits, such as reduced com­plex­i­ty of asset tools, per-ver­tex band­width and stor­age, attribute inter­po­la­tors, trans­form work for skinned mesh­es and last but not least, the pos­si­bil­i­ty to apply nor­mal maps to any pro­ce­du­ral­ly gen­er­at­ed tex­ture coor­di­nates or non-lin­ear defor­ma­tions.

Intermission: Tangents vs Cotangents

The way that nor­mal map­ping is tra­di­tion­al­ly defined is, as I think, flawed, and I would like to point this out with a sim­ple C++ metaphor. Sup­pose we had a class for vec­tors, for exam­ple called Vector3, but we also had a dif­fer­ent class for cov­ec­tors, called Covector3. The lat­ter would be a clone of the ordi­nary vec­tor class, except that it behaves dif­fer­ent­ly under a trans­for­ma­tion (EDIT 2018: see this arti­cle for a com­pre­hen­sive intro­duc­tion to the the­o­ry behind cov­ec­tors and dual spaces). As you may know, nor­mal vec­tors are an exam­ple of such cov­ec­tors, so we’re going to declare them as such. Now imag­ine the fol­low­ing func­tion:

Vector3 tangent; Vector3 bitangent; Covector3 normal;   Covector3 perturb_normal( float a, float b, float c ) { return a * tangent + b * bitangent + c * normal; // ^^^^ compile-error: type mismatch for operator + }

The above func­tion mix­es vec­tors and cov­ec­tors in a sin­gle expres­sion, which in this fic­tion­al exam­ple leads to a type mis­match error. If the normal is of type Covector3, then the tangent and the bitangent should be too, oth­er­wise they can­not form a con­sis­tent frame, can they? In real life shad­er code of course, every­thing would be defined as float3 and be fine, or rather not.

Mathematical Compile Error

Unfor­tu­nate­ly, the above mis­match is exact­ly how the ‘tan­gent frame’ for the pur­pose of nor­mal map­ping was intro­duced by the authors of [2]. This type mis­match is invis­i­ble as long as the tan­gent frame is orthog­o­nal. When the exer­cise is how­ev­er to recon­struct the tan­gent frame in the pix­el shad­er, as this arti­cle is about, then we have to deal with a non-orthog­o­nal screen pro­jec­tion. This is the rea­son why in the book I had intro­duced both (which should be called co-tan­gent) and (now it gets some­what sil­ly, it should be called co-bi-tan­gent) as cov­ec­tors, oth­er­wise the algo­rithm does not work. I have to admit that I could have been more artic­u­late about this detail. This has caused real con­fu­sion, cf from gamedev.net:

The dis­crep­an­cy is explained above, as my ‘tan­gent vec­tors’ are real­ly cov­ec­tors. The def­i­n­i­tion on page 132 is con­sis­tent with that of a cov­ec­tor, and so the frame  should be called a cotan­gent frame.

Intermission 2: Blinns Perturbed Normals (History Channel)

In this sec­tion I would like to show how the def­i­n­i­tion of and as cov­ec­tors fol­lows nat­u­ral­ly from Blinns orig­i­nal bump map­ping paper [1]. Blinn con­sid­ers a curved para­met­ric sur­face, for instance, a Bezi­er-patch, on which he defines tan­gent vec­tors and as the deriv­a­tives of the posi­tion with respect to and .

In this con­text it is a con­ven­tion to use sub­scripts as a short­hand for par­tial deriv­a­tives, so he is real­ly say­ing , etc. He also intro­duces the sur­face nor­mal and a bump height func­tion , which is used to dis­place the sur­face. In the end, he arrives at a for­mu­la for a first order approx­i­ma­tion of the per­turbed nor­mal:

I would like to draw your atten­tion towards the terms and . They are the per­pen­dic­u­lars to and in the tan­gent plane, and can be seen as the ‘off­set vec­tors’ that ulti­mate­ly dis­place the nor­mal. They are also cov­ec­tors (why, make the duck test: if it trans­forms like a cov­ec­tor, it is a cov­ec­tor) so adding them to the nor­mal does not raise said type mis­match. If we divide these terms one more time by and flip their signs, we’ll arrive at the ShaderX5 def­i­n­i­tion of and as fol­lows:

where the hat (as in ) denotes the nor­mal­ized nor­mal. can be inter­pret­ed as the nor­mal to the plane of con­stant , and like­wise as the nor­mal to the plane of con­stant . There­fore we have three nor­mal vec­tors, or cov­ec­tors, , and , and they are the a basis of a cotan­gent frame. Equiv­a­lent­ly, and are the gra­di­ents of and , which is the def­i­n­i­tion I had used in the book. The mag­ni­tude of the gra­di­ent there­fore deter­mines the bump strength, a fact that I will dis­cuss lat­er when it comes to scale invari­ance.

A Little Unlearning

The mis­take of many authors is to unwit­ting­ly take and for and , which only works as long as the vec­tors are orthog­o­nal. Let’s unlearn ‘tan­gent’, relearn ‘cotan­gent’, and repeat the his­tor­i­cal devel­op­ment from this per­spec­tive: Peer­cy et al. [2] pre­com­putes the val­ues and (the change of bump height per change of tex­ture coor­di­nate) and stores them in a tex­ture. They call it ‘nor­mal map’, but is a real­ly some­thing like a ‘slope map’, and they have been rein­vent­ed recent­ly under the name of deriv­a­tive maps. Such a slope map can­not rep­re­sent hor­i­zon­tal nor­mals, as this would need an infi­nite slope to do so. It also needs some ‘bump scale fac­tor’ stored some­where as meta data. Kil­gard [3] intro­duces the mod­ern con­cept of a nor­mal map as an encod­ed rota­tion oper­a­tor, which does away with the approx­i­ma­tion alto­geth­er, and instead goes to define the per­turbed nor­mal direct­ly as

where the coef­fi­cients , and are read from a tex­ture. Most peo­ple would think that a nor­mal map stores nor­mals, but this is only super­fi­cial­ly true. This idea of Kil­gard was, since the unper­turbed nor­mal has coor­di­nates , it is suf­fi­cient to store the last col­umn of the rota­tion matrix that would rotate the unper­turbed nor­mal to its per­turbed posi­tion. So yes, a nor­mal map stores basis vec­tors that cor­re­spond to per­turbed nor­mals, but it real­ly is an encod­ed rota­tion oper­a­tor. The dif­fi­cul­ty starts to show up when nor­mal maps are blend­ed, since this is then an inter­po­la­tion of rota­tion oper­a­tors, with all the com­plex­i­ty that goes with it (for an excel­lent review, see the arti­cle about Reori­ent­ed Nor­mal Map­ping [5] here).

Solution of the Cotangent Frame

The prob­lem to be solved for our pur­pose is the oppo­site as that of Blinn, the per­turbed nor­mal is known (from the nor­mal map), but the cotan­gent frame is unknown. I’ll give a short revi­sion of how I orig­i­nal­ly solved it. Define the unknown cotan­gents and as the gra­di­ents of the tex­ture coor­di­nates and as func­tions of posi­tion , such that

where is the dot prod­uct. The gra­di­ents are con­stant over the sur­face of an inter­po­lat­ed tri­an­gle, so intro­duce the edge dif­fer­ences , and . The unknown cotan­gents have to sat­is­fy the con­straints

where is the cross prod­uct. The first two rows fol­low from the def­i­n­i­tion, and the last row ensures that and have no com­po­nent in the direc­tion of the nor­mal. The last row is need­ed oth­er­wise the prob­lem is under­de­ter­mined. It is straight­for­ward then to express the solu­tion in matrix form. For ,

and anal­o­gous­ly for with .

The above result looks daunt­ing, as it calls for a matrix inverse in every pix­el in order to com­pute the cotan­gent frame! How­ev­er, many sym­me­tries can be exploit­ed to make that almost dis­ap­pear. Below is an exam­ple of a func­tion writ­ten in GLSL to cal­cu­late the inverse of a 3×3 matrix. A sim­i­lar func­tion writ­ten in HLSL appeared in the book, and then I tried to opti­mize the hell out of it. For­get this approach as we are not going to need it at all. Just observe how the adju­gate and the deter­mi­nant can be made from cross prod­ucts:

mat3 inverse3x3( mat3 M ) { // The original was written in HLSL, but this is GLSL, // therefore // - the array index selects columns, so M_t[0] is the // first row of M, etc. // - the mat3 constructor assembles columns, so // cross( M_t[1], M_t[2] ) becomes the first column // of the adjugate, etc. // - for the determinant, it does not matter whether it is // computed with M or with M_t; but using M_t makes it // easier to follow the derivation in the text mat3 M_t = transpose( M ); float det = dot( cross( M_t[0], M_t[1] ), M_t[2] ); mat3 adjugate = mat3( cross( M_t[1], M_t[2] ), cross( M_t[2], M_t[0] ), cross( M_t[0], M_t[1] ) ); return adjugate / det; }

We can sub­sti­tute the rows of the matrix from above into the code, then expand and sim­pli­fy. This pro­ce­dure results in a new expres­sion for . The deter­mi­nant becomes , and the adju­gate can be writ­ten in terms of two new expres­sions, let’s call them and (with read as ‘perp’), which becomes

As you might guessed it, and are the per­pen­dic­u­lars to the tri­an­gle edges in the tri­an­gle plane. Say Hel­lo! They are, again, cov­ec­tors and form a prop­er basis for cotan­gent space. To sim­pli­fy things fur­ther, observe:

• The last row of the matrix is irrel­e­vant since it is mul­ti­plied with zero.
• The oth­er matrix rows con­tain the per­pen­dic­u­lars ( and ), which after trans­po­si­tion just mul­ti­ply with the tex­ture edge dif­fer­ences.
• The per­pen­dic­u­lars can use the inter­po­lat­ed ver­tex nor­mal instead of the face nor­mal , which is sim­pler and looks even nicer.
• The deter­mi­nant (the expres­sion ) can be han­dled in a spe­cial way, which is explained below in the sec­tion about scale invari­ance.

Tak­en togeth­er, the opimized code is shown below, which is even sim­pler than the one I had orig­i­nal­ly pub­lished, but still high­er qual­i­ty:

mat3 cotangent_frame( vec3 N, vec3 p, vec2 uv ) { // get edge vectors of the pixel triangle vec3 dp1 = dFdx( p ); vec3 dp2 = dFdy( p ); vec2 duv1 = dFdx( uv ); vec2 duv2 = dFdy( uv );   // solve the linear system vec3 dp2perp = cross( dp2, N ); vec3 dp1perp = cross( N, dp1 ); vec3 T = dp2perp * duv1.x + dp1perp * duv2.x; vec3 B = dp2perp * duv1.y + dp1perp * duv2.y;   // construct a scale-invariant frame float invmax = inversesqrt( max( dot(T,T), dot(B,B) ) ); return mat3( T * invmax, B * invmax, N ); }

Scale invariance

The deter­mi­nant was left over as a scale fac­tor in the above expres­sion. This has the con­se­quence that the result­ing cotan­gents and are not scale invari­ant, but will vary inverse­ly with the scale of the geom­e­try. It is the nat­ur­al con­se­quence of them being gra­di­ents. If the scale of the geomtery increas­es, and every­thing else is left unchanged, then the change of tex­ture coor­di­nate per unit change of posi­tion gets small­er, which reduces and sim­i­lar­ly in rela­tion to . The effect of all this is a dimin­ished per­tu­ba­tion of the nor­mal when the scale of the geom­e­try is increased, as if a height­field was stretched.

Obvi­ous­ly this behav­ior, while total­ly log­i­cal and cor­rect, would lim­it the use­ful­ness of nor­mal maps to be applied on dif­fer­ent scale geom­e­try. My solu­tion was and still is to ignore the deter­mi­nant and just nor­mal­ize and to whichev­er of them is largest, as seen in the code. This solu­tion pre­serves the rel­a­tive lengths of and , so that a skewed or stretched cotan­gent space is sill han­dled cor­rect­ly, while hav­ing an over­all scale invari­ance.

Non-perspective optimization

As the ulti­mate opti­miza­tion, I also con­sid­ered what hap­pens when we can assume and . This means we have a right tri­an­gle and the per­pen­dic­u­lars fall on the tri­an­gle edges. In the pix­el shad­er, this con­di­tion is true when­ev­er the screen-pro­jec­tion of the sur­face is with­out per­spec­tive dis­tor­tion. There is a nice fig­ure demon­strat­ing this fact in [4]. This opti­miza­tion saves anoth­er two cross prod­ucts, but in my opin­ion, the qual­i­ty suf­fers heav­i­ly should there actu­al­ly be a per­spec­tive dis­tor­tion.

Putting it together

To make the post com­plete, I’ll show how the cotan­gent frame is actu­al­ly used to per­turb the inter­po­lat­ed ver­tex nor­mal. The func­tion perturb_normal does just that, using the back­wards view vec­tor for the ver­tex posi­tion (this is ok because only dif­fer­ences mat­ter, and the eye posi­tion goes away in the dif­fer­ence as it is con­stant).

vec3 perturb_normal( vec3 N, vec3 V, vec2 texcoord ) { // assume N, the interpolated vertex normal and // V, the view vector (vertex to eye) vec3 map = texture2D( mapBump, texcoord ).xyz; #ifdef WITH_NORMALMAP_UNSIGNED map = map * 255./127. - 128./127.; #endif #ifdef WITH_NORMALMAP_2CHANNEL map.z = sqrt( 1. - dot( map.xy, map.xy ) ); #endif #ifdef WITH_NORMALMAP_GREEN_UP map.y = -map.y; #endif mat3 TBN = cotangent_frame( N, -V, texcoord ); return normalize( TBN * map ); }
varying vec3 g_vertexnormal; varying vec3 g_viewvector; // camera pos - vertex pos varying vec2 g_texcoord;   void main() { vec3 N = normalize( g_vertexnormal );   #ifdef WITH_NORMALMAP N = perturb_normal( N, g_viewvector, g_texcoord ); #endif   // ... }

The green axis

Both OpenGL and Direc­tX place the tex­ture coor­di­nate ori­gin at the start of the image pix­el data. The tex­ture coor­di­nate (0,0) is in the cor­ner of the pix­el where the image data point­er points to. Con­trast this to most 3-D mod­el­ing pack­ages that place the tex­ture coor­di­nate ori­gin at the low­er left cor­ner in the uv-unwrap view. Unless the image for­mat is bot­tom-up, this means the tex­ture coor­di­nate ori­gin is in the cor­ner of the first pix­el of the last image row. Quite a dif­fer­ence!
An image search on Google reveals that there is no dom­i­nant con­ven­tion for the green chan­nel in nor­mal maps. Some have green point­ing up and some have green point­ing down. My artists pre­fer green point­ing up for two rea­sons: It’s the for­mat that 3ds Max expects for ren­der­ing, and it sup­pos­ed­ly looks more nat­ur­al with the ‘green illu­mi­na­tion from above’, so this helps with eye­balling nor­mal maps.

Sign Expansion

The sign expan­sion deserves a lit­tle elab­o­ra­tion because I try to use signed tex­ture for­mats when­ev­er pos­si­ble. With the unsigned for­mat, the val­ue 0.5 can­not be rep­re­sent­ed exact­ly (it’s between 127 and 128). The signed for­mat does not have this prob­lem, but in exchange, has an ambigu­ous encod­ing for -1 (can be either -127 or -128). If the hard­ware is inca­pable of signed tex­ture for­mats, I want to be able to pass it as an unsigned for­mat and emu­late the exact sign expan­sion in the shad­er. This is the ori­gin of the seem­ing­ly odd val­ues in the sign expan­sion.

In Hindsight

The orig­i­nal arti­cle in ShaderX5 was writ­ten as a proof-of-con­cept. Although the algo­rithm was test­ed and worked, it was a lit­tle expen­sive for that time. Fast for­ward to today and the pic­ture has changed. I am now employ­ing this algo­rithm in real-life projects for great ben­e­fit. I no longer both­er with tan­gents as ver­tex attrib­ut­es and all the asso­ci­at­ed com­plex­i­ty. For exam­ple, I don’t care whether the COLLADA exporter of Max or Maya (yes I’m rely­ing on COLLADA these days) out­put usable tan­gents for skinned mesh­es, nor do I both­er to import them, because I don’t need them! For the artists, it doesn’t occur to them that an aspect of the asset pipeline is miss­ing, because It’s all nat­ur­al: There is a geom­e­try, there are tex­ture coor­di­nates and there is a nor­mal map, and just works.

Take Away

There are no ‘tan­gent frames’ when it comes to nor­mal map­ping. A tan­gent frame which includes the nor­mal is log­i­cal­ly ill-formed. All there is are cotan­gent frames in dis­guise when the frame is orthog­o­nal. When the frame is not orthog­o­nal, then tan­gent frames will stop work­ing. Use cotan­gent frames instead.

[1] James Blinn, “Sim­u­la­tion of wrin­kled sur­faces”, SIGGRAPH 1978
http://research.microsoft.com/pubs/73939/p286-blinn.pdf

[2] Mark Peer­cy, John Airey, Bri­an Cabral, “Effi­cient Bump Map­ping Hard­ware”, SIGGRAPH 1997
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.42.4736

[3] Mark J Kil­gard, “A Prac­ti­cal and Robust Bump-map­ping Tech­nique for Today’s GPUs”, GDC 2000
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.537

[4] Chris­t­ian Schüler, “Nor­mal Map­ping with­out Pre­com­put­ed Tan­gents”, ShaderX 5, Chap­ter 2.6, pp. 131 – 140

[5] Col­in Bar­ré-Brise­bois and Stephen Hill, “Blend­ing in Detail”,

96 thoughts on “Followup: Normal Mapping Without Precomputed Tangents”

1. Out of inter­est, what impli­ca­tions does this have for tan­gent-space cal­cu­la­tions when you *bake* nor­mal-maps (for exam­ple, inside 3dsmax).
Wouldn’t you have to also use this method when cre­at­ing the orig­i­nal nor­mal-map?

• Hi MoP, absolute­ly, you’re cor­rect in your assump­tion.

If your nor­mal map is the result of sam­pling high poly geom­e­try, then the nor­mal map sam­pler should use the same assump­tions about tan­gent space than the in the engine, if you want high­est fideli­ty. This is gen­er­al­ly true, whether you use pre­com­put­ed tan­gents or not.

If on the oth­er hand you have a paint­ed nor­mal map, let’s say, a gener­ic rip­ple tex­ture, this just fol­lows what­ev­er tex­ture map­ping is applied to the mesh. If this map­ping is sheared or stretched, and so vio­lates the square patch assump­tion, the algo­rithm from above is still able to light it cor­rect­ly.

• Thanks for the fast reply, Chris­t­ian!
Absolute­ly, that makes sense... so I’m won­der­ing if, for “per­fect” results, if peo­ple are bak­ing their nor­mal maps in 3dsmax or Maya or XNor­mal or what­ev­er, those pro­grams will need to have their tan­gent space cal­cu­la­tions updated/reworked to match the in-game cal­cu­la­tion?
I’m guess­ing that cur­rent­ly the 3dsmax cal­cu­la­tion is not exact­ly synced with the method you describe here... or am I wrong?

• You would be sur­prised to know that the method in 3dsMax or any­where else is most like­ly not synced with any engine, even when using the clas­sic method. There is an entire the­sis devot­ed to this top­ic (look for Mikkelsen here). If you want such per­fec­tion, then the engine should have its own tools for bak­ing nor­mal maps because only then it is going to know what their shaders do. 3dsMax can nev­er know this!

• Yep, that makes sense - in fact that’s what the Doom 3 engine (id Tech 4) did to ensure the nor­mal maps were per­fect ... they import­ed LWOs into the engine and baked the nor­mal-maps using the game engine so that the cal­cu­la­tions would match up per­fect­ly.
I haven’t seen any oth­er engine (released pub­li­cal­ly) do that yet, but I work in game devel­op­ment and I know that we have been try­ing to solve this prob­lem for years with no sat­is­fac­to­ry results. Cur­rent­ly the best method seems to be to mod­i­fy a third-par­ty pro­gram (eg. XNor­mal) to cal­cu­late tan­gent space in exact­ly the same way as the tar­get engine cal­cu­lates it, as this is the only way to ensure the nor­mal-maps will be absolute­ly cor­rect.

2. I am con­fused to what space your TBN matrix the nor­mal from the nor­mal map tex­ture con­verts! You need this in order to do light­ing cal­cu­la­tions. Should I trans­form the light vec­tor and view vec­tor with the TBN matrix, or leave them in cam­era space?

Do you assume in the frag­ment shad­er that g_vertexnormal, g_viewvector are in the ver­tex shad­er mul­ti­plied with the gl_NormalMatrix and gl_ModelViewMatrix respec­tive­ly? Oth­er­wise how do you account for the trans­for­ma­tions on the object.

Thanks

• Sor­ry I meant in the pre­vi­ous post “light vec­tor” and “eye vec­tor” instead of view vec­tor for the fist para­graph.

• Hi cinepi­vates,
the TBN matrix that I build in the pix­el shad­er is the trans­form from cotan­gent space to world space. After the vec­tor from the nor­mal map is mul­ti­plied with the TBN matrix, it is a world space nor­mal vec­tor. This is due to the fact that both the inter­po­lat­ed ver­tex nor­mal and the view vec­tor are sup­plied in world space by the ver­tex shad­er (not shown). Alter­na­tive­ly, if the ver­tex shad­er sup­plies these vec­tors in eye space, the pix­el shad­er should con­struct the TBN matrix to con­vert into eye space instead. So you can choose your way.

• Hi
Since you are mul­ti­ply­ing per pix­el with a mat3 (TBNxNor­mal) on the pix­el shad­er have you mea­sured any per­for­mance decrease com­pared to the more tra­di­tion­al method of pre­com­put­ed Tan­gents where you usu­al­ly only trans­form in the ver­tex shad­er the light,eye vec­tors into tan­gent space using the TBN matrix.

• I don’t use the TBN in the ver­tex shad­er. That is a thing of the past when there was pix­el shad­er mod­el 1.x. It pre­vents you to use world space con­stants in the pix­el shad­er. For exam­ple, a reflec­tion vec­tor to look up a world space envi­ron­ment map. So the matrix mul­ti­ply by TBN in the pix­el shad­er has always been there for me.

3. Hi,I just won­der­ing if light­ing cal­cu­la­tions can be done in tan­gent space.My tex­tures could not match the Square Patch Assumption,but I don’t want to lose tan­gent space lighting,since some tech­niques are par­tic­u­lar designed for that.

• Hi April,
this is tricky, as light­ing in tan­gent space does only work if it is orthog­o­nal. Think about it: the angle between any vec­tors (say, the angle between and ) will change under a non-orthog­o­nal trans­for­ma­tion. So dot(N,L) is going to result in a dif­fer­ent val­ue, depend­ing on which space it is in. If you can live with that, then take the inverse-trans­pose of the TBN matrix (this is in essence what you get when you do the non-per­spec­tive opti­miza­tion men­tioned in the post), and use that to trans­form , , and any oth­er vec­tor you need into tan­gent space, and do the light­ing com­pu­ta­tions there.

I man­aged to orthog­o­nal­ize the matrix and then used the inverse-​trans­pose one to do light­ing in world space, the result became a mess.I checked the orgin matrix, and found out the matrix can not be inversed where UV mir­rors. The deter­mi­nant() func­tion returns 0.
Still look­ing for rea­son.

• I’m not sure I can fol­low your argu­ment. If you want to do light­ing in world space, then you don’t need to change any­thing. If you want to do light­ing in tan­gent space instead (which only yields sim­i­lar results if the tan­gent space is rough­ly orthog­o­nal), you’d need a matrix to con­vert your light, view, etc vec­tors into tan­gent space. Edit: For this you need the inverse of the TBN. You can get the inverse-tran­pose already very sim­ply by ignor­ing the two cross prod­ucts and using dp2perp = dp1 and dp1perp = dp2. Then the trans­pose of this would be the inverse of the TBN (ignor­ing scale). In the shad­er you don’t need to explic­it­ly trans­pose, you can just mul­ti­ply with the vec­tors from the left (eg vector * matrix instead of transpose(matrix) * vector). Then you can orthog­o­nal­ize this matrix if you want, but this won’t help much if the tan­gent space is not orthog­o­nal to begin with. I don’t under­stand why you are will­ing to go though such hoops instead of sim­ply doing the light­ing in world space.

4. Fan­tas­tic arti­cle! I switched my engine from using pre­cal­cu­lat­ed tan­gents to the method your describe - very easy to imple­ment. I’m using WebGL, but I’m pre­sum­ing my find­ings will cor­re­spond to what OpenGL + GLES pro­gram­mers see. On desk­top, I don’t real­ly notice much of a per­for­mance dif­fer­ence (if any­thing, the tan­gent-less approach is slight­ly slow­er). But on mobile (tried on both Galaxy Nexus and Nexus 7), this method rough­ly twice as slow as using pre­com­put­ed tan­gents. Seems dFdx and dFdy are par­tic­u­lar­ly slow on mobile GPUs. Just cut­ting out those calls takes my FPS from ~24FPS to ~30FPS. So although I think this method is ele­gant to the extreme, I’m not sure it’s fast enough to be the bet­ter option (at least, not on mobile). I real­ly hope some­one can con­vince me oth­er­wise though! 🙂

• Hi Will, thanks for shar­ing. Of course those addi­tion­al ~14 shad­er instruc­tions for cotangent_frame() are not free, espe­cial­ly not on mobile (which is like 2005 desk­top, the time when the arti­cle was orig­i­nal­ly writ­ten). For me today, this cost is invis­i­ble com­pared to all the oth­er things that are going on, like mul­ti­ple lights and shad­ows and so forth. On the newest archi­tec­tures like the NVidia Fer­mi, it could already be a per­for­mance win to go with­out pre­com­put­ed tan­gents, due to the rea­sons men­tioned in the intro­duc­tion. But while that is nice, the main rea­son I use the method is the boost in pro­duc­tiv­i­ty.

5. For me the equa­tions in this are screwed up. The are ran­dom LaTeX things in the images (Delta­math­bf). Same with Safari, Chrome and Fire­fox.

Would be nice if you could fix that.

6. Some­thing wrong with for­mu­las. Maybe math plu­g­in is bro­ken?

7. Hal­lo there, I have fixed the math for­mu­lae in the post. It turned out to be an incom­pat­i­bil­i­ty between two plu­g­ins, and that got all back­slash­es eat­en. Sor­ry for that! Should there be a bro­ken for­mu­la that I have over­looked, just drop a line.

8. So glad I found your blog. Though I am an artist these more tech­ni­cal insights real­ly help me in under­stand­ing what to com­mu­ni­cate to our coders to estab­lish cer­tain looks.

9. Hi, I want to ask you to elab­o­rate on 2 fol­low­ing moments. First of all I would like to know why cross(N,Pv)/length(N)=gradient(u) and cross(Pu, N)/length(N)=gradient(v). I just can not find any math­e­mat­i­cal deriva­tion of this fact. Also it is unclear for me why deltaU = dot(T, deltaP), and how this fol­lows from def­i­n­i­tion of gra­di­ent. Oth­er­wise I found this arti­cle very excit­ing and my exper­i­ments showed that imple­men­ta­tion works very well and is a good drop-in replace­ment for con­ven­tion­al tan­gent basis.

• Hi Mykhai­lo,
thanks for shar­ing your expe­ri­ence. Indeed, I did not explain why , and that may not be obvi­ous, so here it goes: The gra­di­ent vec­tor is always per­pen­dic­u­lar to the iso-sur­face (aka. “lev­el-set”). In a skewed 2-D coor­di­nate sys­tem, the iso-line of one coor­di­nate is sim­ply the oth­er coor­di­nate axis! So the gra­di­ent vec­tor for the tex­ture coor­di­nate must be per­pen­dic­u­lar to the axis, and vice ver­sa. (The scale fac­tor makes it such that the over­all length equals the rate of change, which is depen­dent on the assump­tion that .) Your sec­ond ques­tion is also relat­ed to the fact that the gra­di­ent vec­tor is per­pen­dic­u­lar to the iso-sur­face. If a posi­tion delta is made par­al­lel to the iso-sur­face, then the tex­ture coor­di­nate doesn’t change, because in this case the dot prod­uct is zero.

• Thank you for an answer, now every­thing is a bit clear­er. Still was able to ful­ly fig­ure it out only after I under­stood that para­metri­sa­tion of u, v is lin­ear in the plane of every tri­an­gle. For some rea­son I just missed that fact. And after you men­tioned that gra­di­ent is per­pen­dic­u­lar to iso-line every­thing made sense. And now it make sense why change of u is pro­jec­tion of posi­tion delta onto gra­di­ent vec­tor.

10. Hi Chris­t­ian, thanks for shar­ing!

I am work­ing on an enhanced ver­sion of the Cry­tek-Spon­za scene and I had prob­lems with my per-pix­el nor­malmap­ping shad­er based on pre­com­put­ed tan­gents. I had most sur­faces light­ed cor­rect­ly, but in some cas­es they weren’t. It turned out that - unknown to me - some faces had flipped UVs (which is not that uncom­mon) and there­fore the cotan­gent frame was messed up, because the inverse of the tan­gent was used to cal­cu­late the binor­mal as the crossprod­uct with the ver­tex nor­mal.

Now, I replaced it with your approach and every­thing is alright; it works instant­ly and all errors van­ished. Awe­some!! 🙂

11. Hi,

Does the com­pu­ta­tion still work if the mesh nor­mal is in view space, and the g_viewvector = vec3(0, 0, -1) ?

Cheers 🙂

• Yes, but you must pro­vide the ver­tex posi­tion in view space also. The view vec­tor is used as a proxy to dif­fer­en­ti­ate the ver­tex posi­tion, there­fore a con­stant view vec­tor will not do.

• Ah ok, I just nor­mal­ized the view posi­tion and its work­ing per­fect­ly

Thanks!

12. Hi,
Your idea seem to be very inter­est­ing.
But I’ve tried it in real scene, and found a glitch - if uv (tex­ture coor­di­nates) are mir­rored, nor­mal is also become mir­rored.

Is there any way to fix it?

• The nor­mal itself should not be flipped, only the tan­gents. If the tex­ture coor­di­nate is mir­rored, then the tan­gent should reverse sign and sim­i­lar with and the tan­gent.

13. In the code you show in the arti­cle, you’re using -V as the posi­tion for tak­ing deriv­a­tives. But V is nor­mal­ized per-pix­el, so your posi­tion deriv­a­tives effec­tive­ly have their radi­al com­po­nent (toward/away from the cam­era) pro­ject­ed out. Doesn’t this cre­ate some arti­facts when the sur­face is at a glanc­ing angle to the cam­era?

(You also lose scale infor­ma­tion, but since you’re nor­mal­iz­ing to make it scale-invari­ant any­way, I sup­pose this does not mat­ter.)

• Hi Nathan the arti­cle doesn’t men­tion it explic­it­ly but the view vec­tor is meant to be passed unnor­mal­ized from ver­tex to pix­elshad­er. The nor­mal­iza­tion should hap­pen after it has been used to com­pute the cotan­gent frame.
(EDIT: I cor­rect­ed the nor­mal­iza­tion mis­take in the exam­ple code, now code + text are in agree­ment)

14. Hel­lo Chris­t­ian,

What can be done with the tex­co­ords if we sam­ple from 2 nor­mal maps with dif­fer­ent uv’s

We add them? or? i guess the rate of change will be the same

Exam­ple code is:
float3 normal1 = tex2d(normalmap1, 3*uv1);
float3 normal2 = tex2d(normalmap2, 5*uv2);

float3 nor­mal = nor­mal­ize (lerp(normal1, normal2, 0.5f) );

Remarks: This oper­a­tion is not good blend­ing at all and my ques­tion is, if it was, how i will can use the uv’s?

Best Regards

• float2 uv_for_perturbation_function = lerp(3*uv1, 5*uv2, 0.5f)

Like this?

• Hi Ste­fan,
I assume that the map­ping of uv1 and uv2 is dif­fer­ent. Then you are going to need a tan­gent frame for each, trans­form both nor­mals into a com­mon space (here: world space) and then mix them.

 float3 normalmap1 = tex2d(normalmap1, uv1); float3 normalmap2 = tex2d(normalmap2, uv2); float3x3 TBN1 = cotangent_frame( vertex_normal, -viewvector, uv1 ); float3x3 TBN2 = cotangent_frame( vertex_normal, -viewvector, uv2 ); float3 nor­mal = nor­mal­ize( lerp( mul( TBN1, normalmap1 ), mul( TBN2, normalmap2 ), 0.5f ) ); 

etc
hope this helps

15. Hel­lo,

I try to do impor­tance sam­pling of the Beck­mann Dis­tri­b­u­tion of a Cook Tor­rance BRDF. I use your descrip­tion to cre­ate a TBN-Matrix in order to trans­form the halfvec­tor from tan­gent-space to world-space.
The prob­lem is that I can see every frag­ment (tri­an­gle) of my geom­e­try and not a glossy shad­ed sur­face. I’m not sure if the prob­lem is real­ly con­nect­ed with the TBN-Matrix. Does it work for light­ing cal­cu­la­tions with­out restric­tions? Or do you think I have to search the mis­take some­where else?

• Hal­lo Phil,
the par­tial deriv­a­tives of tex­ture coor­di­nates (the dFdx and dFdy instruc­tions) are con­stant over the sur­face of a tri­an­gle, so both tan­gent direc­tions, and , are going to be faceted (aka ‘flat shad­ed’). How­ev­er the nor­mal vec­tor is not. So unless you use are very low spec­u­lar expo­nent (high RMS slope in case of Beck­mann) it should not visu­al­ly mat­ter.

16. I tried this, but am hav­ing some prob­lems.
I’m using a left-hand­ed coor­di­nate sys­tem and I pro­vide the nor­mal and the view vec­tor in view space. It seems from one of the com­ments, that this should work.
At first I though every is look­ing good, but than I noticed, that the per­turbed nor­mals are not quite right in some cas­es. Light­ing code should be fine, because results with unper­turbed nor­mals look as expect­ed.
If I use the cotan­gent frame to trans­form the view vec­tor into cotan­gent space for par­al­lax map­ping, the results look cor­rect as well.
I’m also using D3D-style UVs where (0,0) is the top left cor­ner, but that should not mat­ter, right?
Any ideas what could wrong?

• Hi Ben­jamin,
as I said in the arti­cle, the tex­ture coor­di­nate ori­gin is at the start of the image array in both OpenGL and D3D so the UV mir­ror­ing must be done in both APIs. If your coor­di­nate sys­tem is left hand­ed, you’ll need to negate both and , that’s all.

17. I’ve tried imple­ment­ing this, but found that you get a faceted appear­ance on the world space nor­mal, pre­sum­ably due to the deriv­a­tives being per-tri­an­gle. Am I doing some­thing wrong, or is this a lim­i­ta­tion of this tech­nique?

• Hi James,
as some oth­er peo­ple have com­ment­ed, the tan­gen­tial direc­tions are faceted, due to fact tha par­tial deriv­a­tives (dFdx) of tex­ture coor­di­nates are con­stant per tri­an­gle. How­ev­er for the nor­mal direc­tion the inter­po­lat­ed nor­mal is used so the faceting can only appears if there is a dif­fer­ence in the UV gra­di­ents from one tri­an­gle to the next.

18. Could you please post your ver­tex shad­er code? I’m doing some­thing dumb wrong and I’ve been try­ing to get this to work all day.

• Hi Rob,
there is real­ly noth­ing to the ver­tex shad­er, just trans­forms. Below is an exam­ple (transforms[0] is the mod­el-to-world trans­form and transforms[1] is world-to-clip).

 uniform mat4 transforms[2]; uniform vec4 camerapos; varying vec2 texcoord; varying vec3 vertexnormal; varying vec4 viewvector;

void main()
{
vec4 P = transforms[0] * gl_Vertex;
gl_Position = transforms[1] * P;
tex­co­ord = gl_MultiTexCoord0.xy;
ver­texnor­mal =
( transforms[0] * vec4( gl_Normal, 0. ) ).xyz;
viewvec­tor = cam­er­a­pos - P;
}

19. Ohhh, I was hop­ing my prob­lem might be in the ver­tex shad­er code but that’s what I have.

The prob­lem I have is that if I use the nor­mal from perturb_normal, the light­ing rotates with the mod­el so it is always the same side of the mod­el that is lit (beau­ti­ful­ly) no mat­ter how the mod­el is ori­ent­ed rel­a­tive to the light source. If I light the mod­el with just the inter­po­lat­ed ver­tex nor­mal I get light­ing which works as expect­ed.

I wrote out the details of what I’m doing at http://onemanmmo.com/index.php?cmd=newsitem&comment=news.1.158.0 I’m going to have to look at this again tomor­row to see if I can fig­ure out what it is I’m doing wrong. Thank you for your help.

20. (This is a fol­low-up to my pre­vi­ous post, but I can’t find out how to reply to your reply.)

I noticed you say­ing
“Hi Nathan the arti­cle doesn’t men­tion it explic­itly but the view vec­tor is meant to be passed unnor­mal­ized from ver­tex to pix­elshader. The nor­mal­iza­tion should hap­pen after it has been used to com­pute the cotan­gent frame.” In one of the replies.
In the code how­ev­er the view vec­tor is nor­mal­ized before it is passed to per­turb­Nor­mal(). Isn’t this con­tra­dic­to­ry?

The rea­son I’m ask­ing is that I’m still strug­gling to get this work­ing for view space nor­mals. If the nor­malmap only con­tains (0, 0, 1), the final per­turbed nor­mal is unchanged and every­thing works as expect­ed, as the nor­mal effec­tive­ly is not per­turbed at all.
Per­turb­ing by (0, 1, 0) how­ev­er gives me strange results for instance. It is espe­cial­ly notice­able at graz­ing angles. Look­ing at a plane there is a ver­ti­cal line where the nor­mal sud­den­ly flips dras­ti­cal­ly although the sur­face nor­mal did not change at all.
I tried flip­ping B and T etc., but the “flip­ping” effect is not imme­di­ate­ly relat­ed to this.

• Hi Ben­jamin,
thanks for the sug­ges­tion, that’s a seri­ous gotcha, and I cor­rect­ed it.
The code that I post­ed should pass g_viewvector into the func­tion perturb_normal.
If you fol­low the argu­ment in the arti­cle it should be obvi­ous: perturb_normal offi­cial­ly wants the ver­tex posi­tion, but any con­stant off­set to that is going to can­cel when tak­ing the deriv­a­tive. There­fore, the view vec­tor can be passed in as a sur­ro­gate of the ver­tex posi­tion, but this is only true of the unnor­mal­ized view vec­tor.

Have you tried that?

21. Hi,
I’m new to shaders, but I have to imple­ment some fea­tures in a webGL project (based on this tem­plate: http://learningwebgl.com/lessons/lesson14/index.html)
I have already imple­ment­ed a spec­u­lar map, but here are some new things and anoth­er shad­er lan­guage.

What I actu­al­ly want to ask:
- where comes the ‘map­Bump’ from? Is it already imple­ment­ed in GLSL (and webGL)?
- How does ‘map = map * 255./127. - 128./127.;’ work? What means the ‘.’ after each num­ber?

• Nev­er­mind!
In the end, every­thing was self-explan­ing.
Also I just got it work­ing.
The webGL shad­er lan­guage seems to be a bit imma­ture which caused many issues, but final­ly, I got it work­ing.

Thanks for the great tuto­r­i­al!

• Hal­lo Basti,
WebGL is mod­eled after OpenGL ES (and not Desk­top OpenGL), so there may be some incom­pat­i­bil­i­ties. But the shad­ing lan­guage as such should be the same GLSL. Of course, the name ‘map­Bump’ is only an exam­ple and you can use any old name. You should be able to leave out a lead­ing or trail­ing zero in float­ing point num­bers, so you can write 2. instead of 2.0 etc.

22. Hi Chris­t­ian,
Thank you for writ­ing this arti­cle to explain the con­cept of the cotan­gent frame, and also for your atten­tion to imple­men­ta­tion effi­cien­cy details.

I’m still hazy on a num­ber of things though, espe­cial­ly these two:
1.) In the Inter­mis­sion 2 sec­tion, you write:
T = cross(N, Pv)/length(N)^2 = gradient(u)
B = cross(Pu, N)/length(N)^2 = gradient(v)
You seem to imply that if the tan­gent frame is orthog­o­nal, T = Pu and B = Pv, and the dif­fer­ences only occur when the frame has skew. How­ev­er, I’m con­fused about the con­ven­tions you’re using: If we assume cross prod­ucts fol­low the right-hand rule, and if we assume Pu points right and Pv points up, then T = -Pu and B = -Pv for an orthog­o­nal tan­gent frame. In oth­er words, T = Pu and B = Pv only holds for orthog­o­nal tan­gent frames if either:
a.) We’re using the left-hand rule (thank you Direct3D for ruin­ing con­sis­tent lin­ear alge­bra con­ven­tions for­ev­er)
b.) Pv points down (i.e. the tex­ture ori­gin is in the top-left cor­ner, rather than the bot­tom right cor­ner)
...but not both.
What convention/assumption are you using for the above rela­tion­ships? Are you using the same conventions/assumptions through­out the rest of the math­e­mat­i­cal deriva­tion? Would it be more appro­pri­ate in a ped­a­gog­i­cal sense to swap the order of the cross prod­ucts, or am I miss­ing some­thing impor­tant?

2.) I don’t have the back­ground to deeply under­stand the pre­cise dif­fer­ence between a tan­gent frame and a cotan­gent frame. Here is the extent of my back­ground knowl­edge on the sub­ject:
a.) Sur­face nor­mals are covectors/pseudovectors, what­ev­er the heck that means.
b.) Covectors/pseudovectors like nor­mals trans­form dif­fer­ent­ly from ordi­nary vec­tors. Con­sid­er matrix M, col­umn vec­tor v, and nor­mal [col­umn] vec­tor n in the same coor­di­nate frame as v. If you use mul(M, v) to trans­form v into anoth­er coor­di­nate frame, you need to use mul(transpose(inverse(M)), n) to trans­form the nor­mal vec­tor. This reduces to mul(M, n) if M is orthog­o­nal, but oth­er­wise the inverse-trans­pose oper­a­tion is nec­es­sary to main­tain the nor­mal vector’s prop­er­ties in the des­ti­na­tion frame (specif­i­cal­ly, to keep the nor­mal vec­tor per­pen­dic­u­lar to the sur­face). I’ve tak­en this for grant­ed for some time, but I don’t have a deep under­stand­ing of why the inverse-trans­pose is the mag­ic solu­tion for trans­form­ing nor­mal vec­tors, and I believe this might be why I’m hav­ing trou­ble under­stand­ing the nature of the cotan­gent frame.

Now, if I were to cre­ate a “tra­di­tion­al” TBN matrix from deriv­a­tives, I’d do it rough­ly like this (no opti­miza­tion, for clarity...and for­give me if some Cg-like con­ven­tions slip through):
vec3 dp1 = dFdx( p );
vec3 dp2 = dFdy( p );
vec2 duv1 = dFdx( uv );
vec2 duv2 = dFdy( uv );
// set up a lin­ear sys­tem to solve for the TBN matrix:
vec3 p_mat = mat3(dp1, dp2, N);
vec3 uv_mat = mat3(duv1, duv2, vec3(0.0, 0.0, 1.0));
// TBN trans­forms [reg­u­lar, non-nor­mal] vec­tors from
// tan­gent space to world­space, so:
// p_mat = mul(TBN, uv_mat)
// there­fore, solve as:
mat3 TBN = mul(p_mat, inverse(uv_mat));
// exam­ple trans­forms:
vec3 arbitrary_worldspace_vector = mul(TBN, arbitrary_tangent_space_vector);
vec3 worldspace_normal = mul(transpose(inverse(TBN)), normal_map_val);

First, a quick ques­tion: Due to the inverse-trans­pose oper­a­tion, this bla­tant­ly inef­fi­cient “tra­di­tion­al TBN” solu­tion should work cor­rect­ly even for non-orthog­o­nal TBN matri­ces, cor­rect?

Any­way, it fol­lows from the nature of the tan­gent-to-world­space trans­for­ma­tion that p_mat = mul(TBN, uv_mat) above. After all, the whole point of the world­space edges is that they’re the uv-space edges trans­formed from tan­gent-space to world­space.

For this rea­son, I still share some of the con­fu­sion of the poster at http://www.gamedev.net/topic/608004-computing-tangent-frame-in-pixel-shader/. You men­tioned the dis­crep­an­cy has to do with your TBN matrix being a cotan­gent frame com­posed of cov­ec­tors, but I’m not yet clear on the impli­ca­tions of that.

How­ev­er, I noticed some­thing inter­est­ing that I’m hop­ing will make every­thing fall into place: You trans­form your nor­mal vec­tor *direct­ly* from tan­gent-space to world­space using your cotan­gent frame. That is, you just do some­thing like:
vec3 worldspace_normal = mul(cotangent_TBN, normal_map_val);

Is the your cotan­gent frame TBN matrix sim­ply the inverse-trans­pose of the tra­di­tion­al tan­gent frame TBN matrix? If so, and cor­rect me if I’m wrong here...if you want­ed to trans­form ordi­nary vec­tors like L and V from world­space to tan­gent-space (for e.g. par­al­lax occlu­sion map­ping), you would sim­ply do:
tangent_space_V = mul(transpose(cotangent_TBN), worldspace_V);

If I’m under­stand­ing all this cor­rect­ly, con­struct­ing your cotan­gent frame is inher­ent­ly more effi­cient than the tra­di­tion­al TBN for­mu­la­tion for non-orthog­o­nal matri­ces: It lets us trans­form [co]tangent-space nor­mals to world­space nor­mals with a sim­ple TBN mul­ti­pli­ca­tion (instead of an inverse-trans­pose TBN mul­ti­pli­ca­tion), and it lets us trans­form ordi­nary vec­tors from world­space t [co]tangent-space with a trans­pose-TBN mul­ti­pli­ca­tion instead of an inverse-TBN mul­ti­pli­ca­tion.

Do I have this cor­rect, or is the cotan­gent frame matrix some­thing entire­ly dif­fer­ent from what I am now think­ing?

• Hi Michael,
that’s a hand full of a com­ment so i’m try­ing my best to answer.

(1)
Good obser­va­tion! The for­mu­la which says has the cross prod­uct back­wards, and I should cor­rect it in the arti­cle. Note that the code in the shad­er does the right thing, e.g. it says dp2perp = cross( dp2, N ), since my deriva­tion is based on the assump­tion that behaves the way I describe it.

So since I took the for­mu­la out of Blinns Paper, I checked it to see what he has to says about his cross prod­uct busi­ness — and in one of his draw­ings, the vec­tor is indeed in the oppo­site direc­tion as :

(2)
Co-vec­tors are a fan­cy name for “coor­di­nate func­tions”, i.e. plane equa­tions (as in ). The behave like any old vec­tor space and so they’re called the coor­di­nates of a co-vec­tor, while the are the coor­di­nates of an ordi­nary vec­tor.

The ordi­nary vec­tor tells you “the x axis is in this direc­tion”. The co-vec­tor tells you the func­tion for the x-coor­di­nate.

If you have a matrix of col­umn basis vec­tors, then the rows of the inverse matrix are the co-vec­tors, the plane equa­tions that give the coor­di­nate func­tions.

If you stick with the dis­tinc­tion to treat co-vec­tors as rows, you don’t need any of this “inverse trans­pose” busi­ness - just the ordi­nary inverse will do, and you mul­ti­ply nor­mal vec­tors as rows from the left.

(2b)
If your TBN matrix was con­struct­ed as , then yes, you would need to take the inverse (trans­pose) of that to cor­rect­ly trans­form a nor­mal vec­tor in the gen­er­al case. But the whole point of the arti­cle is to con­struct the TBN matrix as instead, which you can use direct­ly, since it is a co-tan­gent frame!

The author of the post at http://www.gamedev.net/topic/608004-computing-tangent-frame-in-pixel-shader/ uses the aster­isk sym­bol “*” to describe a matrix prod­uct in the first case, and a dot prod­uct in the sec­ond case. He won­ders why, seem­ing­ly, the for­mu­la to com­pute the same thing has the “prod­uct” on dif­fer­ent sides by dif­fer­ent authors. That’s his “dis­crep­an­cy”. The truth is: The for­mu­lae do not com­pute the same thing!

Is the your cotan­gent frame TBN matrix sim­ply the inverse-​trans­pose of the tra­di­tional tan­gent frame TBN matrix?

Yes, you can say that and are the “inverse-trans­pose” of and . The third one, , does not change direc­tion under the inverse-trans­pose oper­a­tion, since it is orthog­o­nal to the oth­er two. And if it is unit length, it will stay unit length.

23. Excel­lent! Thank you so much for tak­ing the time to answer my ques­tions. I had start­ed to sus­pect that the inverse-trans­pose of the Pu|Pv|N matrix was a round­about way of com­put­ing what you com­pute direct­ly, and I’m very glad to hear that’s the case. I still have some learn­ing to do to com­fort­ably manip­u­late cov­ec­tors, but it helps a great deal that the end result is some­thing I recognize...and under­stand­ing cov­ec­tors more thor­ough­ly will final­ly take the “black mag­ic” out of why the inverse-trans­pose of Pu|Pv|N also works (much less effi­cient­ly of course).

24. Pingback: Balancing | Spellcaster Studios

25. Hi there! I’m hav­ing trou­ble using the func­tions dFdx() and dFdy()... tried adding this line:
#exten­sion GL_OES_standard_derivatives : enable
in my shad­er, but it gives me the mes­sage that the exten­sion isn’t sup­port­ed. Do you have any idea about how i’m sup­posed to do this?

26. Hi Chris­t­ian,
I’m research­ing on the run-time gen­er­ate tbn matrix relat­ed top­ics recent­ly, and I found your arti­cle very inter­est­ing. I’m won­der­ing if I want to inte­grate your glsl shad­er with my code, which is writ­ten in directx/hlsl, should I use the -N instead in the final result of cotan­gent frame matrix?(for deal­ing with the Right-hand­ed and left-hand­ed issue). If this is not the solu­tion, what should I do? Thanks.

• Hi Sher­ry
the one gotcha you need to be aware of is that HLSL’s ddy has dif­fer­ent sign than dFdy, due to OpenGL’s win­dow coor­di­nates being bot­tom-to-top (when not ren­der­ing into an FBO, that is). Oth­er than that, it is just syn­tac­ti­cal code con­ver­sion. For test­ing, you can sub­sti­tute N with the face nor­mal gen­er­at­ed by cross­ing dp1 and dp2; that must work in every case, what­ev­er sign con­ven­tion the screen space deriv­a­tives have.

27. Hi Chris­t­ian!
First, thanks for an explana­to­ry arti­cle, it’s great!
I still have a cou­ple of ques­tions. Lets list some facts:
a) You said in response to Michael that transpose(TBN) can be used to trans­form eg. V vec­tor from world-space to tan­gent-space.
b) dFdx and dFdy, and chence dp1/2 and duv1/2 are con­stant over a tri­an­gle.
Based on that facts, can your TBN be com­put­ed in geom­e­try-shad­er on per-tri­an­gle basis, and its trans­pose used to trans­form V, L, Blinn’s half vec­tor, etc. to tan­gent-space, in order to make “clas­sic” light­ning in pix­el-shad­er?
I’ve found some­thing sim­i­lar in: http://www.slideshare.net/Mark_Kilgard/geometryshaderbasedbumpmappingsetup
but there is noth­ing about com­mon tan­gent-basis cal­cu­la­tion in tex­ture bak­ing tool and shad­er. The lat­ter is necce­sary, since sub­sti­tut­ing nor­mal map with flat nor­mal-map (to get sim­ple light­ning) pro­duces faceted look, as in Kilgard’s approach. The sec­ond ques­tion: do you have a plu­g­in for xNor­mal, which can com­pute cor­rect per-tri­an­gle tan­gent basis?

• Hi Andrzej,
yes, you can (and I have) do the same cal­cu­la­tions in the geom­e­try shad­er. The cotan­gent basis is always com­put­ed from a tri­an­gle. In the pix­el shad­er, the ‘tri­an­gle’ is implic­it­ly spanned between the cur­rent pix­el and neigh­bor­ing pix­els. In the geom­e­try shad­er, you use the actu­al tri­an­gle, i.e., dFdx and dFdy are sub­sti­tut­ed with the actu­al edge dif­fer­ences. The rest will be iden­ti­cal.
You can then pass TBN down to the pix­el shad­er, or use it to trans­form oth­er quan­ti­ties and pass these.

Faceting: As I said before, the T and B vec­tors will be faceted, but the N vec­tor can use the inter­po­lat­ed nor­mal to give a smoother look. In prac­tice, if the UV map­ping does not stray too far from the square patch assump­tion, the faceting will be unno­tice­able.

28. Thank You very much, Chris­t­ian!

I imple­ment­ed the tech­nique and wrote XNor­mal plu­g­in, yet, when trans­form­ing gen­er­at­ed tan­gent-space nor­mal map back to object-space (by XNor­mal tool), I didn’t get the expect­ed result. I’ll try to make per-tri­an­gle tbn com­pu­ta­tion order-inde­pen­dent and check the code.

29. Real­ly nice post, I’m work­ing on some ter­rain at the moment for a poject in my spare time.. I came accross your site yes­ter­day while look­ing for an in shad­er bumpmap­ping tech­nique and Imle­ment­ed it right away
I real­ly like the results and per­for­mance also seems good.. but I was won­der­ing about apply­ing such tech­niques to large scenes, I don’t need the shad­er applied to cer­tain regions(black/very dark places).. But my under­stand­ing is that it will be applied to ever pix­el on the ter­rain regard­less of whether I want it lit or not..
This is a bit off top­ic so apolo­gies for that, but in terms on opti­miza­tion, if applied to a black region would glsl avoid pro­cess­ing that or do you have any rec­om­men­da­tions for per­for­mance tweak­ing, conditionals/branching etc.. I under­stand a sten­cil buffer could assist here, but these type of opti­miza­tions are real­ly inter­est­ing, it could make a good post in it’s own right.
Thanks again for your insight­ful arti­cle!

• Hi Cor­mac,
since the work is done per pix­el, the per­for­mance doesn’t depend on the size of the scene, but only on the num­ber of ren­dered pix­els. This is a nice prop­er­ty which is called “out­put-sen­si­tiv­i­ty”, ie. the per­for­mance depends on the size of the out­put, not the size of the input. For large scenes, you want all ren­der­ing to be out­put sen­si­tive, so you do occlu­sion culling and the like.
Branch­ing opti­miza­tions only pay off if the amount of work that is avoid­ed is large, for instance, when skip­ping over a large num­ber of shad­ow map sam­ples. In my expe­ri­ence, the tan­gent space cal­cu­la­tion described here (essen­tial­ly on the order of 10 to 20 shad­er instruc­tions) is not worth the cost of a branch. But if in doubt, just pro­file it!

30. Thanks Chris­t­ian !!
The per­for­mance is actu­al­ly sur­pris­ing­ly good.. The card I’m run­ning on is quite old and FPS drop is not sig­nif­i­cant which is impres­sive.
It’s fun­ny when I applied it to my scene I start­ed get­ting some strange ari­facts in the nor­mals. I didn’t have time too dig into it yet, per­hap my axes are mixed up or some­thing, I’ll check it again tonight.

• If you mean the seams that appear at tri­an­gle bor­ders, these are relat­ed to the fact that dFdx/dFdy of the tex­ture coor­di­nate is con­stant across a tri­an­gle, so there will be a dis­con­tin­u­ous change in the two tan­gent vec­tors when the direc­tion of the tex­ture map­ping changes. This is expect­ed behav­ior, espe­cial­ly if the tex­ture map­ping is sheared/stretched, as is usu­al­ly the case on a pro­ce­dur­al ter­rain. The light­ing will be/should be cor­rect.

31. Hi Chris­t­ian!
I thought about seams and light­ning dis­con­ti­nu­ities and have anoth­er cou­ple of ques­tions in this area. But first some facts.
1. When we con­sid­er a tex­ture with nor­mals in object space there are almost no seams; dis­tor­tions are most­ly relat­ed to nonex­act inter­po­la­tion over an edge in a mesh, whose sides are uncon­nect­ed in the tex­ture (eg. due to dif­fer­ent lenght, angles, etc.)
2. No mat­ter how a tan­gent space is defined (per-pix­el with T,B and N inter­po­lat­ed over tri­an­gle, with only N inter­po­lat­ed, or even con­stant TBN over tri­an­gle), tan­gent-space nor­mals should always decode to object-space coun­ter­parts.

Why it isn’t the case in Your method? I think its because tri­an­gles adja­cent in a mesh are also adja­cent in tex­ture-space. Then, when ren­der­ing an edge, you read the nor­mal from a tex­ture, but you may ‘decode’ it with a wrong tan­gent-space basis - from the oth­er side tri­an­gle (with p=0.5). Things get worse with lin­ear tex­ture sam­pler applied instead of point sam­pler. Note how the trick with ‘the same’ tan­gent-space on both sides of the edge works well in this sit­u­a­tion.

So, the ques­tion: shoudn’t all tri­an­gles in a mesh be uncon­nect­ed in the tex­ture space for Your method to work? (It sholdn’t be a prob­lem to code an addi­tion­al ‘tex­ture-break­er’ for a tool-chain.)

• Hi Andrzej,
have a look at the very first posts in this thread. There is a dis­cus­sion about the dif­fer­ence between “paint­ed nor­mal maps” and “baked nor­mal maps”.

For paint­ed nor­mal maps, the method is in prin­ci­ple, cor­rect. The light­ing is always exact­ly faith­ful to the implied height map, the slope of which may change abrupt­ly at a tri­an­gle bor­der, depend­ing on the UV map­ping. That’s just how things are.

For baked nor­mal maps, where you want a result that looks the same as a high-poly geom­e­try, the bak­ing pro­ce­dure would have to use the same TBN that is gen­er­at­ed in the pix­el shad­er, in order to match per­fect­ly.
This can be done, how­ev­er due to tex­ture inter­po­la­tion, the dis­con­ti­nu­ity can only be approx­i­mat­ed down to the tex­el lev­el. So there would be slight mis­match with­in the tex­el that strad­dles the tri­an­gle bound­ary. If you want to elim­i­nate even that, you need to pay 3 times the num­ber of ver­tices to make the UV atlas for each tri­an­gle sep­a­rate.

In my prac­tice, I real­ly don’t care. We’re have been using what every­one else does: x-nor­mal, crazy bump, sub­stance B2M, etc, and I have yet to receive a sin­gle ‘com­plaint’ from artists. 🙂

32. Some­how I for­got that nor­mal maps can also be paint­ed 🙂
Maybe sep­a­rat­ing tri­an­gles in the tex­ture domain is unnec­es­sary in prac­tice, but as I am teach­ing about nor­mal map­ping, I want to know every aspect of it.

Thanks a lot!

33. Chris­t­ian, please help me with anoth­er issue. In many NM tuto­ri­als over the net, light­ning is done in tan­gent space, i.e., Light and Blinn’s Half are trans­formed into tan­gent space at each ver­tex, to be inter­po­lat­ed over a tri­an­gle. Con­sid­er­ing each TBN forms ortho­nor­mal basis, why this works only for L and not for H? It gives me strange arte­facts, while inter­po­lat­ing ‘plain’ H and trans­form­ing it at each frag­ment by inter­po­lat­ed & nor­mal­ized TBN works well (for sim­plic­i­ty, I con­sid­er world-space object-space).

• Hi Andrzej,
there are some ear­li­er com­ments on this issue. One of the salient points of the arti­cle is that the TBN frame is, in gen­er­al, not ortho­nor­mal. There­fore, dot prod­ucts are not pre­served across trans­forms.

34. Hi Chris­t­ian,
Thanks! But in main case I do have ortho­nor­mal base at each ver­tex (N from mod­el, T com­put­ed by mikk­T­Space and orthog­o­nal­ized by Gram-Schmit method, B as cross prod­uct). Accord­ing to many tuto­ri­als this should work, yet in [2] they say this can be con­sid­ered an approx­i­ma­tion only. Who is right then? I don’t know if there is bug in my code or tri­an­gles in low poly mod­el are too big for such approx­i­ma­tion?

35. If you like cov­ec­tors, I guess you will like Geo­met­ric Alge­bra and Grass­mann Alge­bra 🙂

36. Hi Chris­t­ian,
Very fas­ci­nat­ing arti­cle and it looks to be exact­ly what I was look­ing for. Unfor­tu­nate­ly, I can’t seem to get this work­ing in HLSL: I must be doing some­thing real­ly stu­pid but I can’t seem to fig­ure it out. I’m doing my light­ing cal­cu­la­tions in view space but hav­ing read through the com­ments, it shouldn’t affect the cal­cu­la­tions regard­less?

Here’s the “port” of your code: http://pastebin.com/cDr1jnPb

Thanks!

• Hi Peter,
when trans­lat­ing from GLSL to HLSL you must observe two things:
1. Matrix order (row- vs col­umn major). The mat3(...) con­struc­tor in GLSL assem­bles the matrix col­umn-wise.
2. The dFdy and ddy have dif­fer­ent sign, because the coor­di­nate sys­tem in GL is bot­tom-to-top.

37. hi, l’d like to ask if this method can be used in the ward anisotrop­ic light­ing equa­tion, which is using tan­gent and binor­mal direct­ly.

I found the result to be facet because the tan­gent and binor­mal are facet, is that means I can’t use your method in this sit­u­a­tion? sor­ry about my bad eng­lish..

• Hey Maval,

I just came across this same post when I was inves­ti­gat­ing the usage of com­put­ing our tan­gents in the frag­ment shad­er.

Since T and B are faceted with this tech­nique, it can­not be used with an anisotrop­ic brdf, so long as the anisotropy depends on the tan­gent frame.

I also do not think it’s pos­si­ble to cor­rect for this with­out addi­tion­al infor­ma­tion non-local from the tri­an­gle.

Indy

• Hi Maval,
the faceting of the tan­gents is not very not­i­ca­ble in prac­tice, so I would just give it a try and see what hap­pens. It all depends on how strong the tan­gents are curved with­in the spe­cif­ic UV map­ping at hand.

38. Wow thank you very much for this. I was sup­posed to imple­ment nor­mal map­ping in my PBR ren­der­er but couldn’t get over the fact that those co-tan­gent and bi-co-tan­gent were a huge pain in the back to trans­fer to the ver­tex shad­er (with much data dupli­cat­ed). This is so much bet­ter, thank you :D.

39. Hi Chris­t­ian. I was try­ing to pro­duce sim­i­lar for­mu­las in a dif­fer­ent way and some­how the result is dif­fer­ent and does not work. As a chal­lenge for myself I’m try­ing to find the error in my approach but can not.

As a basis of my approach do deduce gra­di­ent du/dp, I assume that both tex­ture coor­di­nates u and world-point p on the tri­an­gle are func­tions of screen coor­di­nates (sx, sy). So to dif­fer­en­ti­ate du(p(sx, sy))/dp(sx, sy) I use rule of par­tial deriv­a­tives:
du/dp = du/dsx * dsx/dp + du/dsy * dsy/dp. From here (du/dsx, du/dsy) is basi­cal­ly (duv1.x, duv2.x) in your code.

To com­pute dsx/dp, I try to inverse the deriv­a­tives: dsx/dp = 1 / (dsx/dp), which is equal to (1 / dFdx(p.x), 1 / dFdx(p.y), 1 / dFdx(p.z)), but some­how that does not work because the result is dif­fer­ent from dp2perp in your code. Can you clar­i­fy why?

• Your terms and are vec­tor val­ued and togeth­er form a 2-by-3 Jaco­bian matrix, that con­tains the con­tri­bu­tion of each com­po­nent of the world space posi­tion to a change in screen coor­di­nate.

To get the inverse, you’d need to inverse that matrix, but you can­not do that because it’s not a square matrix. The prob­lem is under­de­ter­mined. Aug­ment the Jaco­bian with a vir­tu­al 3rd screen coor­di­nate (the screen z coor­di­nate, or depth coor­di­nate) to and equate that to zero, because it must stay con­stant for the solu­tion to lie on the screen plane. Et voila, then you would have arrived at an exact­ly equiv­a­lent prob­lem for­mu­la­tion as that which is described in the sec­tion ‘solu­tion to the co-tan­gent frame’.

40. In my library, I have a debug shad­er that lets me visu­al­ize the nor­mal, tan­gent, and (recon­struct­ed) bi-tan­gent in the tra­di­tion­al “pre­com­put­ed ver­tex tan­gents” sce­nario.

I’m try­ing to work out what the equiv­a­lent to the “per ver­tex tan­gent” is with a cal­cu­lat­ed TBN. The sec­ond and third row vec­tors don’t seem to match very close­ly with the orig­i­nal tan­gen­t/bi-tan­gent.

• Sor­ry, make that the first and sec­ond row vec­tor. The third row vec­tor is obvi­ous­ly the orig­i­nal per-ver­tex nor­mal.

• Hi Chuck,
to get some­thing that is like the tra­di­tion­al per-ver­tex tan­gent, you’d have to take the first two columns of the invert­ed matrix.

The tan­gents and the co-tan­gents each answer dif­fer­ent ques­tions. The tan­gent answers what is the change in posi­tion with a change in uv. The co-tan­gents are nor­mals to the planes of con­stant u and v, so they answer what direc­tion is per­pen­dic­u­lar to keep­ing uv con­stant. They’ll be equiv­a­lent as long as the tan­gent frame is orthog­o­nal, but diverge when that is not the case.

41. You do this in a ter­ri­bly round­about way, it’s much eas­i­er to do

”’

vec3 denorm­Tan­gent = dFdx(texCoord.y)*dFdy(vPos)-dFdx(vPos)*dFdy(texCoord.y);
vec3 tan­gent = normalize(denormTangent-smoothNormal*dot(smoothNormal,denormTangent));

vec3 nor­mal = normalize(smoothNormal);
vec3 bitan­gent = cross(normal,tangent);
”’

• Hi devsh,
that would be equiv­a­lent to what the arti­cle describes as the “non-per­spec­tive-opti­miza­tion”. In this case you’re no longer mak­ing a dis­tinc­tion between tan­gents and co-tan­gents. It only works if the tan­gent frame is orthog­o­nal. In case of per-pix­el dif­fer­ences tak­en with dFdx etc, it would only be cor­rect if the mesh is dis­played with­out per­spec­tive dis­tor­tion, hence the name.