determinant for non square matrix do not exist.
However you probably talk about the matrix for 2D operations, they are really 3x3 but the last row is 0, 0, 1 hence it "simplified" as 2x3.
I think the determinant might simply be M11 * M22 - m21 * m12, as sugeested there btw
Reference Source[
^]