The basic algorithm is this: each side of a polygon is defined by the coordinate of the two points of the vertices. You can take each pair of sides of two polygons and examine it. Formally, you represent each side as an equation of the straight line; for another side of another polygon, you have another straight line equation; and then you solve to pair of algebraic equations to find if there is an intersection, which is the point when x and y of two sides are equal. In case you find the intersection, you need to perform an additional check: the location of this point should belong to the line segment of each of the two sides. This is not always the case. You can have the intersection, but it could be found on the "continuation" of one of both segments. You have to do this check for each pair of sides of both shapes.

If you need to learn the related mathematics, you can try to find some publicly accessible material referenced, for example, on this page: http://en.wikipedia.org/wiki/Linear_algebra[^].

But this is not all. You will need to address two difficulties I mentioned in my first paragraph above. What happens when some two segments are "almost parallel" and intersect at some point in the close vicinity of the point sets represented by each of the side? In such cases, the presence of intersection is surprisingly difficult to detect.

To get a general idea, please see http://en.wikipedia.org/wiki/Numerical_stability#Stability_in_numerical_linear_algebra[^].

You can consider using some available mathematics library. You can start, for example, here:

http://en.wikipedia.org/wiki/Comparison_of_linear_algebra_libraries[^],

https://www.quantconnect.com/blog/top-numerical-libraries-for-c[^],

http://www.dotnumerics.com/NumericalLibraries/LinearAlgebra[^].

like this[^]

—SA

Bruno