Hello,
I do not think there is any book or documentation on such a specific subject. Too few people develop CAD or 3D modeling software.
In addition, setting up a good software architecture does not really depend on the type of software itself. The main difficulty here is to be able to think of a good plug-in / API system from the start of the project and then maintain a clean architecture even after adding a lot of new functionalities / features.
Sorry I can't help more.
We are in the process of open sourcing our multi purpose C++ framework (
GitHub - assoria/kigs: Kigs framework is a C++ modular multipurpose cross platform framework.[
^] ) so I would have proposed to use it as a basis for your developments, but the Wiki is still being written.
Good luck for the future.