This is quite a difficult problem, to make the editor the way it is done in the page you referenced. This is done throw low-level graphics which is drawn on a bitmap which is then shown on the page. Drawing is not a problem at all, but logical structure of mathematical expression and translating it into geometry of traditional mathematical notations is a really big work. In you have some basic question, I'll answer, but you cannot expect any advanced solutions.
Pay attention that the same page is supposed to support much easier alternative: MathML (as alpha-version). In this case, all the graphical rendering is already done. HTML5 supports direct embedding of MathML in HTML. As this way assumes that you use a lot of existing MathML functionality, this problem looks order of magnitude more realistic. However, a Quick Answer won't help you to get a quick and easy solution; this is still a lot of serious work. If you still want to start such a project, please see:
http://en.wikipedia.org/wiki/MathML[
^],
http://en.wikipedia.org/wiki/HTML5#Differences_from_HTML.C2.A04.01_and_XHTML.C2.A01.x[
^].
—SA