It's open source, so start by downloading
the source[
^], compile it as it stands to make sure it works before you start changing it, and then examine the code carefully to work out how it works.
Then it's the same as any other project: specify your changes, design, implement, test: repeat until ready for release...
Other than that, it's all up to you! But you might be able to get more specific help on the
UltraVNC forum[
^]