I found this routine sometimes produces incorrect tesselation. When certain points in the polygon winding are colinear, it can skip over them and produce triangles outside of the original polygon.
My fix simply classifies points on the edge of the test triangle as 'inside' instead of 'outside'. This appears to be a stable and beneficial fix so far.
Example Input:
poly normal (0.000f,0.500f,0.866f)
poly points
(0.700f,6.500f,4.071f)
(0.700f,5.500f,3.494f)
(1.700f,5.500f,3.494f)
(1.700f,6.500f,4.071f)
(2.500f,6.500f,4.071f)
(7.300f,1.700f,1.300f)
(11.300f,1.700f,1.300f)
(7.500f,5.500f,3.494f)
(4.300f,5.500f,3.494f)
(4.300f,6.500f,4.071f)
Old Code in function IsPointInside():
pmq2.Vector(m_e1,ntmp); if( (B0=m_N.Dot(ntmp)) <= 0.0 ) return false;
m_e0.Vector(pmq2,ntmp); if( (B1=m_N.Dot(ntmp)) <= 0.0 ) return false;
return ( (m_AB0B1) > 0.0 ? true : false );
New Code:
pmq2.Vector(m_e1,ntmp); if( (B0=m_N.Dot(ntmp)) < 0.0 ) return false;
m_e0.Vector(pmq2,ntmp); if( (B1=m_N.Dot(ntmp)) < 0.0 ) return false;
return ( (m_AB0B1) < 0.0 ? false : true);
Helpful explaination of internal variables:
Edit....
Further testing shows that neither the old or new code correctly and consistently handles coincident and colinear points. (Eg. An example of a coincident point, is a Figure '8' polygons that does not actually cross over, just converges in the middle.)
Example Input:
poly normal (0.000f,0.500f,0.866f)
poly points
(1.700f,6.300f,2.801f)
(4.300f,3.700f,1.300f)
(5.700f,3.700f,1.300f)
(7.000f,5.000f,2.051f)
(8.300f,3.700f,1.300f)
(9.300f,3.700f,1.300f)
(8.000f,5.000f,2.051f)
(7.000f,5.000f,2.051f)
(1.700f,10.300f,5.111f)
This should help more with coincident points: Old code in IsAnyPointInside():
if( ( ip < i  ip > k ) &&
IsPointInside(points[m_nIndex[ip]],points[ik]) )
{
return true;
}
New code:
if( ip < i  ip > k )
{
if( points[m_nIndex[ip]].Equal(points[m_nIndex[i]])
 points[m_nIndex[ip]].Equal(points[m_nIndex[j]])
 points[m_nIndex[ip]].Equal(points[m_nIndex[k]])
)
{
continue;
}
if( IsPointInside(points[m_nIndex[ip]],points[ik]) )
{
return true;
}
}
If topus's suggested change is valid, it should read
if( ((FLT_EPSILON) < m_A && m_A < FLT_EPSILON) 
((FLT_EPSILON) < m_N[0] && m_N[0] < FLT_EPSILON &&
(FLT_EPSILON) < m_N[1] && m_N[1] < FLT_EPSILON &&
(FLT_EPSILON) < m_N[2] && m_N[2] < FLT_EPSILON ))
return degenerate;
Also note that you can help the algorithm by removing 'junk' before processing. Eg. project 3D points onto plane (of which normal is used). Remove degeneracies like coincident and colinear points. Snapping and merging points may help, but could lead to badsnaps that cause self intersections.
Edit: I have found another case which I believe shows a flaw in the algorithm. As 'ears' are clipped off the polygon, the remaining polygon can become self intersecting and fail 'point in poly' test, to produce triangles outside the original shape.
Example input:
poly normal(0,0,1)
(10.52881622f,1.25890017f,0.00000000f)
(10.52881241f,0.24110639f,0.00000000f)
(1.47119045f,0.24110317f,0.00000000f)
(1.47119141f,1.25889719f,0.00000000f)
(4.52881575f,1.25890088f,0.00000000f)
(4.52881479f,3.25889254f,0.00000000f)
(6.08583260f,3.25889111f,0.00000000f)
(6.08583069f,1.25889945f,0.00000000f)
The fix: in function Triangulate() in 'case convex :' in else, after 'RemoveVertex'
i = j;
j = k;
k++;
This allows the point AFTER the removed triangle point to be the start of the next test triangle. This helps with near colinear points that fan out from an otherwise stationary start point.
Edit: There are still issues with the code. The bottom line is that the original algorithm does not handle simple noncovex polys with colinear, but non intersecting segments. The overall algorithm does not handle polygons that intersect in anyway, including at single points, though it often produces a correct or near correct result.
Please comment on this.
