Detecting memory leaks isnt easy, so you need some tools if you really want to do it. I would try at first to contact some people with deep boost knowledege.
Sometimes the task manager is fooling at the first look. Read
here how to find memory leaks. And if you want to dig deeper you may find
here some useful tool.
I personally would try to avoid this work, because the benefits may not pay for the effort. So I would try to do more simpler logging or use other logging tools.