Followup: Normal Mapping Without Precomputed Tangents

This post is a fol­low-up to my 2006 ShaderX5 arti­cle [4] about nor­mal map­ping with­out a pre-com­put­ed tan­gent basis. In the time since then I have refined this tech­nique with lessons learned in real life. For those unfa­mil­iar with the top­ic, the moti­va­tion was to con­struct the tan­gent frame on the fly in the pix­el shad­er, which iron­i­cal­ly is the exact oppo­site of the moti­va­tion from [2]:

Since it is not 1997 any­more, doing the tan­gent space on-the-fly has some poten­tial ben­e­fits, such as reduced com­plex­i­ty of asset tools, per-ver­tex band­width and stor­age, attribute inter­po­la­tors, trans­form work for skinned mesh­es and last but not least, the pos­si­bil­i­ty to apply nor­mal maps to any pro­ce­du­ral­ly gen­er­at­ed tex­ture coor­di­nates or non-lin­ear deformations.

Intermission: Tangents vs Cotangents

The way that nor­mal map­ping is tra­di­tion­al­ly defined is, as I think, flawed, and I would like to point this out with a sim­ple C++ metaphor. Sup­pose we had a class for vec­tors, for exam­ple called Vector3, but we also had a dif­fer­ent class for cov­ec­tors, called Covector3. The lat­ter would be a clone of the ordi­nary vec­tor class, except that it behaves dif­fer­ent­ly under a trans­for­ma­tion (Edit 2018: see this arti­cle for a com­pre­hen­sive intro­duc­tion to the the­o­ry behind cov­ec­tors and dual spaces). As you may know, nor­mal vec­tors are an exam­ple of such cov­ec­tors, so we’re going to declare them as such. Now imag­ine the fol­low­ing function:

Vector3 tangent;
Vector3 bitangent;
Covector3 normal;
 
Covector3 perturb_normal( float a, float b, float c )
{
    return a * tangent +
           b * bitangent + 
           c * normal; 
           // ^^^^ compile-error: type mismatch for operator +
}

The above func­tion mix­es vec­tors and cov­ec­tors in a sin­gle expres­sion, which in this fic­tion­al exam­ple leads to a type mis­match error. If the normal is of type Covector3, then the tangent and the bitangent should be too, oth­er­wise they can­not form a con­sis­tent frame, can they? In real life shad­er code of course, every­thing would be defined as float3 and be fine, or rather not.

Mathematical Compile Error

Unfor­tu­nate­ly, the above mis­match is exact­ly how the ‘tan­gent frame’ for the pur­pose of nor­mal map­ping was intro­duced by the authors of [2]. This type mis­match is invis­i­ble as long as the tan­gent frame is orthog­o­nal. When the exer­cise is how­ev­er to recon­struct the tan­gent frame in the pix­el shad­er, as this arti­cle is about, then we have to deal with a non-orthog­o­nal screen pro­jec­tion. This is the rea­son why in the book I had intro­duced both \vec{T} (which should be called co-tan­gent) and \vec{B} (now it gets some­what sil­ly, it should be called co-bi-tan­gent) as cov­ec­tors, oth­er­wise the algo­rithm does not work. I have to admit that I could have been more artic­u­late about this detail. This has caused real con­fu­sion, cf from gamedev.net:

The dis­crep­an­cy is explained above, as my ‘tan­gent vec­tors’ are real­ly cov­ec­tors. The def­i­n­i­tion on page 132 is con­sis­tent with that of a cov­ec­tor, and so the frame \left(\vec{T}|\vec{B}|\vec{N}\right) should be called a cotan­gent frame.

Intermission 2: Blinns Perturbed Normals (History Channel)

In this sec­tion I would like to show how the def­i­n­i­tion of \vec{T} and \vec{B} as cov­ec­tors fol­lows nat­u­ral­ly from Blinn’s orig­i­nal bump map­ping paper [1]. Blinn con­sid­ers a curved para­met­ric sur­face, for instance, a Bezi­er-patch, on which he defines tan­gent vec­tors \vec{p}_u and \vec{p}_v as the deriv­a­tives of the posi­tion \vec{p} with respect to u and v.

In this con­text it is a con­ven­tion to use sub­scripts as a short­hand for par­tial deriv­a­tives, so he is real­ly say­ing \vec{p}_u = {\partial\vec{p} \over \partial u}, etc. He also intro­duces the sur­face nor­mal \vec{N} = \vec{p}_u \times \vec{p}_v and a bump height func­tion f, which is used to dis­place the sur­face. In the end, he arrives at a for­mu­la for a first order approx­i­ma­tion of the per­turbed normal:

    \[\vec{N}' \simeq \vec{N} + \frac{f_u \vec{N} \times \vec{p}_v + f_v \vec{p}_u \times \vec{N}}{|\vec{N}|} ,\]

I would like to draw your atten­tion towards the terms \vec{N} \times \vec{p}_v and \vec{p}_u \times \vec{N}. They are the per­pen­dic­u­lars to \vec{p}_u and \vec{p}_v in the tan­gent plane, and togeth­er form a vec­tor basis for the dis­place­ments f_u and f_v. They are also cov­ec­tors (this is easy to ver­i­fy as they behave like cov­ec­tors under trans­for­ma­tion) so adding them to the nor­mal does not raise said type mis­match. If we divide these terms one more time by |\vec{N}| and flip their signs, we’ll arrive at the ShaderX5 def­i­n­i­tion of \vec{T} and \vec{B} as follows:

    \begin{align*} \vec{T} &= -\frac{\vec{N} \times \vec{p}_v}{|\vec{N}|^2} = \nabla u, & \vec{B} &= -\frac{\vec{p}_u \times \vec{N}}{|\vec{N}|^2} = \nabla v, \end{align*}

    \[\vec{N}' \simeq \hat{N} - f_u \vec{T} - f_v \vec{B} ,\]

where the hat (as in \hat{N}) denotes the nor­mal­ized nor­mal. \vec{T} can be inter­pret­ed as the nor­mal to the plane of con­stant u, and like­wise \vec{B} as the nor­mal to the plane of con­stant v. There­fore we have three nor­mal vec­tors, or cov­ec­tors, \vec{T}, \vec{B} and \vec{N}, and they are the a basis of a cotan­gent frame. Equiv­a­lent­ly, \vec{T} and \vec{B} are the gra­di­ents of u and v, which is the def­i­n­i­tion I had used in the book. The mag­ni­tude of the gra­di­ent there­fore deter­mines the bump strength, a fact that I will dis­cuss lat­er when it comes to scale invariance.

A Little Unlearning

The mis­take of many authors is to unwit­ting­ly take \vec{T} and \vec{B} for \vec{p}_u and \vec{p}_v, which only works as long as the vec­tors are orthog­o­nal. Let’s unlearn ‘tan­gent’, relearn ‘cotan­gent’, and repeat the his­tor­i­cal devel­op­ment from this per­spec­tive: Peer­cy et al. [2] pre­com­putes the val­ues f_u and f_v (the change of bump height per change of tex­ture coor­di­nate) and stores them in a tex­ture. They call it ’nor­mal map’, but is a real­ly some­thing like a ’slope map’, and they have been rein­vent­ed recent­ly under the name of deriv­a­tive maps. Such a slope map can­not rep­re­sent hor­i­zon­tal nor­mals, as this would need an infi­nite slope to do so. It also needs some ‘bump scale fac­tor’ stored some­where as meta data. Kil­gard [3] intro­duces the mod­ern con­cept of a nor­mal map as an encod­ed rota­tion oper­a­tor, which does away with the approx­i­ma­tion alto­geth­er, and instead goes to define the per­turbed nor­mal direct­ly as

    \[\vec{N}' = a \vec{T} + b \vec{B} + c \hat{N} ,\]

where the coef­fi­cients a, b and c are read from a tex­ture. Most peo­ple would think that a nor­mal map stores nor­mals, but this is only super­fi­cial­ly true. The idea of Kil­gard was, since the unper­turbed nor­mal has coor­di­nates (0,0,1), it is suf­fi­cient to store the last col­umn of the rota­tion matrix that would rotate the unper­turbed nor­mal to its per­turbed posi­tion. So yes, a nor­mal map stores basis vec­tors that cor­re­spond to per­turbed nor­mals, but it real­ly is an encod­ed rota­tion oper­a­tor. The dif­fi­cul­ty starts to show up when nor­mal maps are blend­ed, since this is then an inter­po­la­tion of rota­tion oper­a­tors, with all the com­plex­i­ty that goes with it (for an excel­lent review, see the arti­cle about Reori­ent­ed Nor­mal Map­ping [5] here).

Solution of the Cotangent Frame

The prob­lem to be solved for our pur­pose is the oppo­site as that of Blinn, the per­turbed nor­mal is known (from the nor­mal map), but the cotan­gent frame is unknown. I’ll give a short revi­sion of how I orig­i­nal­ly solved it. Define the unknown cotan­gents \vec{T} = \nabla u and \vec{B} = \nabla v as the gra­di­ents of the tex­ture coor­di­nates u and v as func­tions of posi­tion \vec{p}, such that

    \begin{align*} \mathrm{d} u &= \vec{T} \cdot \mathrm{d} \vec{p} , & \mathrm{d} v &= \vec{B} \cdot \mathrm{d} \vec{p} , \end{align*}

where \cdot is the dot prod­uct. The gra­di­ents are con­stant over the sur­face of an inter­po­lat­ed tri­an­gle, so intro­duce the edge dif­fer­ences \Delta u_{1,2}, \Delta v_{1,2} and \Delta \vec{p_{1,2}}. The unknown cotan­gents have to sat­is­fy the constraints

    \begin{align*} \Delta u_1 &= \vec{T} \cdot \Delta \vec{p_1} , & \Delta v_1 &= \vec{B} \cdot \Delta \vec{p_1} , \\ \Delta u_2 &= \vec{T} \cdot \Delta \vec{p_2} , & \Delta v_2 &= \vec{B} \cdot \Delta \vec{p_2} , \\ 0 &= \vec{T} \cdot \Delta \vec{p_1} \times \Delta \vec{p_2} , & 0 &= \vec{B} \cdot \Delta \vec{p_1} \times \Delta \vec{p_2} , \end{align*}

where \times is the cross prod­uct. The first two rows fol­low from the def­i­n­i­tion, and the last row ensures that \vec{T} and \vec{B} have no com­po­nent in the direc­tion of the nor­mal. The last row is need­ed oth­er­wise the prob­lem is under­de­ter­mined. It is straight­for­ward then to express the solu­tion in matrix form. For \vec{T},

    \[\vec{T} = \begin{pmatrix} \Delta \vec{p_1} \\ \Delta \vec{p_2} \\ \Delta \vec{p_1} \times \Delta \vec{p_2} \end{pmatrix}^{-1} \begin{pmatrix} \Delta u_1 \\ \Delta u_2 \\ 0 \end{pmatrix} ,\]

and anal­o­gous­ly for \vec{B} with \Delta v.

Into the Shader Code

The above result looks daunt­ing, as it calls for a matrix inverse in every pix­el in order to com­pute the cotan­gent frame! How­ev­er, many sym­me­tries can be exploit­ed to make that almost dis­ap­pear. Below is an exam­ple of a func­tion writ­ten in GLSL to cal­cu­late the inverse of a 3×3 matrix. A sim­i­lar func­tion writ­ten in HLSL appeared in the book, and then I tried to opti­mize the hell out of it. For­get this approach as we are not going to need it at all. Just observe how the adju­gate and the deter­mi­nant can be made from cross products:

mat3 inverse3x3( mat3 M )
{
    // The original was written in HLSL, but this is GLSL, 
    // therefore
    // - the array index selects columns, so M_t[0] is the 
    //   first row of M, etc.
    // - the mat3 constructor assembles columns, so 
    //   cross( M_t[1], M_t[2] ) becomes the first column
    //   of the adjugate, etc.
    // - for the determinant, it does not matter whether it is
    //   computed with M or with M_t; but using M_t makes it
    //   easier to follow the derivation in the text
    mat3 M_t = transpose( M ); 
    float det = dot( cross( M_t[0], M_t[1] ), M_t[2] );
    mat3 adjugate = mat3( cross( M_t[1], M_t[2] ),
                          cross( M_t[2], M_t[0] ),
                          cross( M_t[0], M_t[1] ) );
    return adjugate / det;
}

We can sub­sti­tute the rows of the matrix from above into the code, then expand and sim­pli­fy. This pro­ce­dure results in a new expres­sion for \vec{T}. The deter­mi­nant becomes \left| \Delta \vec{p_1} \times \Delta \vec{p_2} \right|^2, and the adju­gate can be writ­ten in terms of two new expres­sions, let’s call them \Delta \vec{p_1}_\perp and \Delta \vec{p_2}_\perp (with \perp read as ‘perp’), which becomes

    \[\vec{T} = \frac{1}{\left| \Delta \vec{p_1} \times \Delta \vec{p_2} \right|^2} \begin{pmatrix} \Delta \vec{p_2}_\perp \\ \Delta \vec{p_1}_\perp \\ \Delta \vec{p_1} \times \Delta \vec{p_2} \end{pmatrix}^\mathrm{T} \begin{pmatrix} \Delta u_1 \\ \Delta u_2 \\ 0 \end{pmatrix} ,\]

    \begin{align*} \Delta \vec{p_2}_\perp &= \Delta \vec{p_2} \times \left( \Delta \vec{p_1} \times \Delta \vec{p_2} \right) , \\ \Delta \vec{p_1}_\perp &= \left( \Delta \vec{p_1} \times \Delta \vec{p_2} \right) \times \Delta \vec{p_1} . \end{align*}

As you might guessed it, \Delta \vec{p_1}_\perp and \Delta \vec{p_2}_\perp are the per­pen­dic­u­lars to the tri­an­gle edges in the tri­an­gle plane. Say Hel­lo! They are, again, cov­ec­tors and form a prop­er basis for cotan­gent space. To sim­pli­fy things fur­ther, observe:

  • The last row of the matrix is irrel­e­vant since it is mul­ti­plied with zero.
  • The oth­er matrix rows con­tain the per­pen­dic­u­lars (\Delta \vec{p_1}_\perp and \Delta \vec{p_2}_\perp), which after trans­po­si­tion just mul­ti­ply with the tex­ture edge differences.
  • The per­pen­dic­u­lars can use the inter­po­lat­ed ver­tex nor­mal \vec{N} instead of the face nor­mal \Delta \vec{p_1} \times \Delta \vec{p_2}, which is sim­pler and looks even nicer.
  • The deter­mi­nant (the expres­sion \left| \Delta \vec{p_1} \times \Delta \vec{p_2} \right|^2) can be han­dled in a spe­cial way, which is explained below in the sec­tion about scale invariance.

Tak­en togeth­er, the opti­mized code is shown below, which is even sim­pler than the one I had orig­i­nal­ly pub­lished, and yet high­er quality:

mat3 cotangent_frame( vec3 N, vec3 p, vec2 uv )
{
    // get edge vectors of the pixel triangle
    vec3 dp1 = dFdx( p );
    vec3 dp2 = dFdy( p );
    vec2 duv1 = dFdx( uv );
    vec2 duv2 = dFdy( uv );
 
    // solve the linear system
    vec3 dp2perp = cross( dp2, N );
    vec3 dp1perp = cross( N, dp1 );
    vec3 T = dp2perp * duv1.x + dp1perp * duv2.x;
    vec3 B = dp2perp * duv1.y + dp1perp * duv2.y;
 
    // construct a scale-invariant frame 
    float invmax = inversesqrt( max( dot(T,T), dot(B,B) ) );
    return mat3( T * invmax, B * invmax, N );
}

Scale invariance

The deter­mi­nant \left| \Delta \vec{p_1} \times \Delta \vec{p_2} \right|^2 was left over as a scale fac­tor in the above expres­sion. This has the con­se­quence that the result­ing cotan­gents \vec{T} and \vec{B} are not scale invari­ant, but will vary inverse­ly with the scale of the geom­e­try. It is the nat­ur­al con­se­quence of them being gra­di­ents. If the scale of the geomtery increas­es, and every­thing else is left unchanged, then the change of tex­ture coor­di­nate per unit change of posi­tion gets small­er, which reduces \vec{T} = \nabla u = \left( \frac{\partial u}{\partial x}, \frac{\partial u}{\partial y}, \frac{\partial u}{\partial z} \right) and sim­i­lar­ly \vec{B} in rela­tion to \vec{N}. The effect of all this is a dimin­ished per­tu­ba­tion of the nor­mal when the scale of the geom­e­try is increased, as if a height­field was stretched.

Obvi­ous­ly this behav­ior, while total­ly log­i­cal and cor­rect, would lim­it the use­ful­ness of nor­mal maps to be applied on dif­fer­ent scale geom­e­try. My solu­tion was and still is to ignore the deter­mi­nant and just nor­mal­ize \vec{T} and \vec{B} to whichev­er of them is largest, as seen in the code. This solu­tion pre­serves the rel­a­tive lengths of \vec{T} and \vec{B}, so that a skewed or stretched cotan­gent space is sill han­dled cor­rect­ly, while hav­ing an over­all scale invariance.

Non-perspective optimization

As the ulti­mate opti­miza­tion, I also con­sid­ered what hap­pens when we can assume \Delta \vec{p_1} = \Delta \vec{p_2}_\perp and \Delta \vec{p_2} = \Delta \vec{p_1}_\perp. This means we have a right tri­an­gle and the per­pen­dic­u­lars fall on the tri­an­gle edges. In the pix­el shad­er, this con­di­tion is true when­ev­er the screen-pro­jec­tion of the sur­face is with­out per­spec­tive dis­tor­tion. There is a nice fig­ure demon­strat­ing this fact in [4]. This opti­miza­tion saves anoth­er two cross prod­ucts, but in my opin­ion, the qual­i­ty suf­fers heav­i­ly should there actu­al­ly be a per­spec­tive distortion.

Putting it together

To make the post com­plete, I’ll show how the cotan­gent frame is actu­al­ly used to per­turb the inter­po­lat­ed ver­tex nor­mal. The func­tion perturb_normal does just that, using the back­wards view vec­tor for the ver­tex posi­tion (this is ok because only dif­fer­ences mat­ter, and the eye posi­tion goes away in the dif­fer­ence as it is constant).

vec3 perturb_normal( vec3 N, vec3 V, vec2 texcoord )
{
    // assume N, the interpolated vertex normal and 
    // V, the view vector (vertex to eye)
    vec3 map = texture2D( mapBump, texcoord ).xyz;
#ifdef WITH_NORMALMAP_UNSIGNED
    map = map * 255./127. - 128./127.;
#endif
#ifdef WITH_NORMALMAP_2CHANNEL
    map.z = sqrt( 1. - dot( map.xy, map.xy ) );
#endif
#ifdef WITH_NORMALMAP_GREEN_UP
    map.y = -map.y;
#endif
    mat3 TBN = cotangent_frame( N, -V, texcoord );
    return normalize( TBN * map );
}
varying vec3 g_vertexnormal;
varying	vec3 g_viewvector;  // camera pos - vertex pos
varying vec2 g_texcoord;
 
void main()
{
    vec3 N = normalize( g_vertexnormal );
 
#ifdef WITH_NORMALMAP
    N = perturb_normal( N, g_viewvector, g_texcoord );
#endif
 
    // ...
}

The green axis

Both OpenGL and Direc­tX place the tex­ture coor­di­nate ori­gin at the start of the image pix­el data. The tex­ture coor­di­nate (0,0) is in the cor­ner of the pix­el where the image data point­er points to. Con­trast this to most 3‑D mod­el­ing pack­ages that place the tex­ture coor­di­nate ori­gin at the low­er left cor­ner in the uv-unwrap view. Unless the image for­mat is bot­tom-up, this means the tex­ture coor­di­nate ori­gin is in the cor­ner of the first pix­el of the last image row. Quite a difference!
An image search on Google reveals that there is no dom­i­nant con­ven­tion for the green chan­nel in nor­mal maps. Some have green point­ing up and some have green point­ing down. My artists pre­fer green point­ing up for two rea­sons: It’s the for­mat that 3ds Max expects for ren­der­ing, and it sup­pos­ed­ly looks more nat­ur­al with the ‘green illu­mi­na­tion from above’, so this helps with eye­balling nor­mal maps.

Sign Expansion

The sign expan­sion deserves a lit­tle elab­o­ra­tion because I try to use signed tex­ture for­mats when­ev­er pos­si­ble. With the unsigned for­mat, the val­ue ½ can­not be rep­re­sent­ed exact­ly (it’s between 127 and 128). The signed for­mat does not have this prob­lem, but in exchange, has an ambigu­ous encod­ing for −1 (can be either −127 or −128). If the hard­ware is inca­pable of signed tex­ture for­mats, I want to be able to pass it as an unsigned for­mat and emu­late the exact sign expan­sion in the shad­er. This is the ori­gin of the seem­ing­ly odd val­ues in the sign expansion.

In Hindsight

The orig­i­nal arti­cle in ShaderX5 was writ­ten as a proof-of-con­cept. Although the algo­rithm was test­ed and worked, it was a lit­tle expen­sive for that time. Fast for­ward to today and the pic­ture has changed. I am now employ­ing this algo­rithm in real-life projects for great ben­e­fit. I no longer both­er with tan­gents as ver­tex attrib­ut­es and all the asso­ci­at­ed com­plex­i­ty. For exam­ple, I don’t care whether the COLLADA exporter of Max or Maya (yes I’m rely­ing on COLLADA these days) out­put usable tan­gents for skinned mesh­es, nor do I both­er to import them, because I don’t need them! For the artists, it does­n’t occur to them that an aspect of the asset pipeline is miss­ing, because It’s all nat­ur­al: There is a geom­e­try, there are tex­ture coor­di­nates and there is a nor­mal map, and just works.

Take Away

There are no ‘tan­gent frames’ when it comes to nor­mal map­ping. A tan­gent frame which includes the nor­mal is log­i­cal­ly ill-formed. All there is are cotan­gent frames in dis­guise when the frame is orthog­o­nal. When the frame is not orthog­o­nal, then tan­gent frames will stop work­ing. Use cotan­gent frames instead.


References

[1] James Blinn, “Sim­u­la­tion of wrin­kled sur­faces”, SIGGRAPH 1978
http://research.microsoft.com/pubs/73939/p286-blinn.pdf

[2] Mark Peer­cy, John Airey, Bri­an Cabral, “Effi­cient Bump Map­ping Hard­ware”, SIGGRAPH 1997
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.42.4736

[3] Mark J Kil­gard, “A Prac­ti­cal and Robust Bump-map­ping Tech­nique for Today’s GPUs”, GDC 2000
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.18.537

[4] Chris­t­ian Schüler, “Nor­mal Map­ping with­out Pre­com­put­ed Tan­gents”, ShaderX 5, Chap­ter 2.6, pp. 131 – 140

[5] Col­in Bar­ré-Brise­bois and Stephen Hill, “Blend­ing in Detail”,
http://blog.selfshadow.com/publications/blending-in-detail/

111 Gedanken zu „Followup: Normal Mapping Without Precomputed Tangents

  1. You do this in a ter­ri­bly round­about way, it’s much eas­i­er to do 

    ”’

    vec3 denorm­Tan­gent = dFdx(texCoord.y)*dFdy(vPos)-dFdx(vPos)*dFdy(texCoord.y);
    vec3 tan­gent = normalize(denormTangent-smoothNormal*dot(smoothNormal,denormTangent));

    vec3 nor­mal = normalize(smoothNormal);
    vec3 bitan­gent = cross(normal,tangent);
    ”’

    • Hi devsh,
      that would be equiv­a­lent to what the arti­cle describes as the “non-per­spec­tive-opti­miza­tion”. In this case you’re no longer mak­ing a dis­tinc­tion between tan­gents and co-tan­gents. It only works if the tan­gent frame is orthog­o­nal. In case of per-pix­el dif­fer­ences tak­en with dFdx etc, it would only be cor­rect if the mesh is dis­played with­out per­spec­tive dis­tor­tion, hence the name.

  2. Pingback: Normal and normal mapping – One Line to rule them all

  3. Hel­lo!
    First I want to thank for the algo­rithm, this is a very nec­es­sary thing. He works on my HLSL!
    But there is a bug. When approach­ing the cam­era close to the object, a strong noise of pix­el tex­ture begins. This is a known prob­lem; no one has yet solved it. Do you have any deci­sions on this, have you thought about this?

    • Hi Zagol­s­ki
      This behav­iour is like­ly that the dif­fer­ences between pix­els of the tex­ture coor­di­nate become too small for float­ing point pre­ci­sion and are then round­ed to zero, which leads to a divide by zero down the line.

      Try to elim­i­nate any „half“ pre­ci­sion vari­able that may affect the tex­ture coor­di­nate or view vec­tor, if there are any.

  4. Pingback: Skin Rendering with Texture Space Diffusion – Polysoup Kitchen

  5. Does this approach work with mir­rored uv’s? We’re see­ing some light­ing issues where the light­ing is incor­rect on the mir­rored parts of shapes. Maya knows to flip the nor­mals, but does­n’t know to flip the uv direc­tions. So then the TBN frame looks incorrect.

    • Yes, it should work with mir­rored tex­ture coor­di­nates out of the box. Then the frame spanned by T and B sim­ply changes handedness.

  6. Nice work. It was real­ly sim­ple to inte­grate in my code, the results look very good, and the expla­na­tions are just great!

    I observed, how­ev­er, that the first two col­umn vec­tors of TBN — those that replace the clas­si­cal inter­po­lat­ed pre-ver­tex tan­gents and bitan­gents — are smooth over a tri­an­gle, how­ev­er, they are not smooth across tri­an­gle edges. Note that the tan­gents and bitan­gents are smooth across tri­an­gle edges. 

    Any thoughts on whether this is a prob­lem? To be clear: The final light­ing does­n’t seem to look wrong and the nor­mal used for light­ing do not appear to have those dis­con­ti­nu­ities, but I am still curi­ous on your expert’s opin­ion on that.

    Thanks

    Quirin

    • Hal­lo Quirin,
      as you have noticed, the pseudovec­tors T and B are faceted. This stems from the fact that all deltas (\Delta \vec{p}_1, etc) are con­stant for all pix­els of a giv­en tri­an­gle. If you browse through the ear­li­er com­ment his­to­ry then you’ll find many more dis­cus­sions about this fact. In my opin­ion this has nev­er been a prob­lem in prac­tice. As you notice, the inter­po­lat­ed nor­mal is used for N, so that is smooth.

  7. Hi, as a bit of a 3D gen­er­al­ist, and rel­a­tive­ly new to shaders at that, I can appre­ci­ate both the thor­ough math analy­sis and the sheer prac­ti­cal­i­ty of in-shad­er com­pu­ta­tion from noth­ing else than nor­mals and UVs. It is a bless­ing to have found this at an ear­ly point of my jour­ney, while putting togeth­er my first scene view­er app. Thank you for shar­ing your insight the way you did.

    That said, I am not writ­ing just for the sake of praise. I do have a cou­ple of ques­tions, of the “did I get this right?” kind, which may prove use­ful to oth­er peo­ple who try this code in a “toy” project.

    1) In the GLSL main() exam­ple, g_vertexnormal and g_viewvector are in world space, right? (Because there would­n’t be a “cam­era pos” in view or frus­tum space.) Sup­pos­ing the pro­gram pass­es raw ver­tices and nor­mals (in mod­el space) to the shad­er, along with a bunch of matri­ces, g_vertexnormal would need to be trans­formed as a cov­ec­tor, right? (as in, N = normalize(transpose(inverse(mat3(M))) * g_vertexnormal ), M being the 4x4 mod­el matrix such that vertex_pos_world = mat3(M * vec4(vertex_pos_model,1))
    And of course the adjoint mod­el matrix would only be dif­fer­ent from the plain mod­el matrix if non-uni­form scal­ing is involved (right?).

    2) Despite N being smooth across tri­an­gle edges (in the­o­ry at least), I am see­ing some sub­tle light­ing dis­con­ti­nu­ities, espe­cial­ly when I add spec­u­lar to the mix. My under­stand­ing is that the code is real­ly fine, and I did­n’t mess up imple­ment­ing it either, it’s just OpenGL’s default tex­ture fil­ter­ing. If the code is imple­ment­ed from scratch (e.g., in a “toy project” such as mine), then dis­con­ti­nu­ities at edges are to be expect­ed at some view­ing angles (despite the algorithm’s accu­ra­cy), and will only dis­ap­pear once I set up prop­er mipmap­ping and anisotrop­ic fil­ter­ing. Right?

    Thanks again,
    Sergey

    • Hal­lo Sergey,
      thanks for your comments!
      For 1)
      Both vec­tors g_vertexnormal and g_viewvector must be in the same space, and then the cal­cu­lat­ed nor­mal is in that space too.
      — Rea­son not to do it in mod­el space: pos­si­ble non-uni­form scale, don’t!
      — Rea­son not to do it in view space: while pos­si­ble, need to trans­form all envi­ron­men­tal light data from world into view space -> arkward
      So my rec­om­men­da­tion is world space ori­en­ta­tion, but camera-relative.
      For 2)
      As was writ­ten as response to many oth­er com­menters: Your code is prob­a­bly fine. The arti­facts are there because the deriv­a­tives of the tex­ture coor­di­nates are con­stant across a tri­an­gle. So the T and B vec­tors do not change over the sur­face of a tri­an­gle. There­fore, the stronger your nor­mal map, the stronger the faceted arti­facts, because T and B are only rel­e­vant for mod­i­fy­ing N accord­ing to the nor­mal map, and the direc­tion in which this mod­i­fi­ca­tion is done may change abrubtly across tri­an­gle bor­ders. Mipmap­ping will reduce the nor­mal map inten­si­ty by aver­ag­ing every­thing out.

  8. Hi Chris­t­ian,

    Checked on the page yes­ter­day. Glad to see my post reg­is­tered after all, and thanks for the answer!

    1) Thanks for the con­fir­ma­tion. I have tried a bunch of dif­fer­ent coor­di­nates while debug­ging, and indeed cam­era-rel­a­tive world coor­di­nates make per­fect sense for your algo­rithm. (My first shad­er attempt was in view space because of some light­ing code exam­ples that were writ­ten that way, but as you say work in view space would become less and less intu­itive for com­plex lighting. 

    2) In your pre­vi­ous reply to Quirin (about a year ago), you made it sound like your algo­rithm nev­er leads to faceted looks in prac­tice, and that a smooth inter­po­lat­ed nor­mal N is some­how suf­fi­cient to ensure a smooth per­turbed nor­mal PN, even when T ad B are dis­con­tin­u­ous across an edge. But now you make a valid point about how a dis­con­tin­u­ous T or B will be per­turb­ing the nor­mal by dif­fer­ent amounts and in dif­fer­ent direc­tions (for the same nor­mal map pix­el) on both sides of an edge. So now I am real­ly curi­ous which it is! In the gen­er­al case, the lat­ter is prob­a­bly true, i.e., there will always be some mesh­es that have arti­facts because of dis­con­tin­u­ous T and B (sad!). But clear­ly for a sym­met­ric enough edge the dis­con­tin­u­ous con­tri­bu­tions can can­cel out, so that the per­turbed nor­mal is still smooth and facet-less. What, then, is the con­di­tion for such a nice com­pen­sa­tion to occur, and could­n’t the algo­rithm be adjust­ed so that it hap­pens more often?

    It turns out that, in my case, there actu­al­ly *was* a bug: my spec­u­lar code involves a nor­mal­ized eye vec­tor, so I end­ed up pass­ing that to perturb_normal instead of a non-nor­mal­ized one. While debug­ging, I cre­at­ed sev­er­al test shapes with well-behaved UVs (antiprisms, tes­se­lat­ed cubes, etc), and it was a relief to see the facets dis­ap­pear once I fixed that nor­mal­iza­tion mis­take. (I will try stronger nor­mal maps on those test shapes, and maybe write up some shape-spe­cif­ic math, but for sym­met­ric enough edges/UVs I don’t expect dis­con­tin­u­ous T and B to cause faceted­ness per se.) I also have a lot of mesh­es (third-par­ty FBX files, most­ly) for which facets can be seen in many areas, but part of that is prob­a­bly due to bad UVs/normals/geometry. I’ll try and pin­point the sim­plest pos­si­ble mesh for which your algo­rithm has sig­nif­i­cant “faceting”, and then report back.

    Cheers,
    Sergey

    • Hal­lo Sergey,
      sor­ry for not check­ing more often. But legit com­ments will get through eventually.
      As for 2) this con­di­tion was orig­i­nal­ly termed by Peer­cy et al as the “square patch assump­tion”: The UV map­ping should look like a square grid, with­out shear, and with­out non-uni­form scale. Con­se­quent­ly then, the TBN frame is orthog­o­nal, mod­u­lo a scale fac­tor. Then no vis­i­ble change occurs across tri­an­gles. While not exact­ly real­iz­able in prac­tice, “good” work on part of the artist may get close to that ide­al, and is a desire­able qual­i­ty also for oth­er reasons.
      Cheers!
      Christian

  9. Hi, thanks for sharing.
    Is it nec­es­sary to use view vec­tor as p in cotangent_frame? Would it be suf­fi­cient to use world position?

    • Hal­lo Tomáš,
      the view vec­tor and the world vec­tor only dif­fer by the the eye vec­tor, which is a con­stant. There­fore, either view vec­tor or world posi­tion may be used for \mathbf{p}, since their deriv­a­tives are the same.

      Using the view vec­tor should behave bet­ter numer­i­cal­ly because the world posi­tion may have large mag­ni­tudes that can­cel bad­ly when the deriv­a­tive func­tion is applied.

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert

Please answer the following anti-spam test

Which thing makes "tick-tock" and if it falls down, the clock is broken?

  1.    pencil
  2.    watch
  3.    table
  4.    chair
  5.    ruler