Click here to Skip to main content
Click here to Skip to main content

Tagged as

Memory Leak Caused by Fragmentation

, 23 Feb 2006
Rate this:
Please Sign up or sign in to vote.
This article shows the effect of Memory Fragmentation and provides a generic object pool class.

Introduction

Some days ago, I finished fixing a bug called "memory leak" that I think I'll never forget during my career.

My program was about a real-time monitor that read data from a server and viewed graphs. My program met the "memory leak" problem when data throughput was too heavy. I remember, every 100 ms the program had to read about 2 Kb data. And then, after only a day, my program ran out of virtual memory although the private bytes still looked pretty good on the Performance tool.

I tried to detect memory leak by using a lot of tools, from free of charge tools to trade tools as I was used to do before, but none of the tools could find out any memory leak.

Background

Problem

Now, let's consider the problem:

//The following was a global buffer
CPtrArray g_oBuffer;
//...

//Any time new data arrived, I allocated
//a new memory and added it to the buffer
MyData* oData = new MyData();
g_oBuffer.Add(oData);
//...

//When old data wasn't needed anymore, I removed it
MyData* oData = (MyData*)g_oBuffer.GetAt(0);
g_oBuffer.RemoveAt(0);
delete oData;
//...

Everything looked fine and that was the reason why my program's private bytes looked stable. But unfortunately, after only a day, I met a memory exception and saw the virtual bytes reach the 2 GB limitation.

The Performance monitor looked like below:

Sample Image

Solution

Instead of allocating memory one by one, I decided to pre-allocate a large amount of memory into an object pool and use it when necessary. Then, my code became like this:

//Any time new data arrived, I got a pre-allocated memory
//from the object pool and added it to the buffer

//MyData* oData = new MyData();
MyData* oData = (MyData*)MyObjectPool::GetNew();
g_oBuffer.Add(oData);
//...

//When old data wasn't needed anymore, I removed it
//and returned the memory to the object pool
MyData* oData = g_oBuffer.RemoveAt(0);
//MyData* oData = g_oBuffer.RemoveAt(0);
MyObjectPool::Delete(oData);
//...

After applying this solution, I was impressed with the result as below:

Sample Image

Using the code

In order to provide a generic object pool, I re-implemented the object pool as a template class. With this article, I'd just like to show a real experience on Memory Fragmentation and a solution which has been applied successfully. Besides, I only did a few tests on my code. Therefore, it may still have some bugs remaining. If anyone could find out a bug, please send a feedback to me. I will really appreciate it.

To use this generic object pool, all you have to do is:

#define StrPool CObjectPoolImpl<CString>

//...
//Create a new pool
StrPool MyPool(10); //Size in Mega byte
//...
//Create a new CString object
CString* pStr = MyPool.getNewObj();
//...
//Delete CString object when it's not used any more
MyPool.deleteObj(pStr);
//...
//Empty the pool
MyPool.deleteAllObj();

References

From the bottom of my heart, I would like to say "thanks" to the author Danny Kalev who has written this article. This one was very short but cool enough to describe what a memory fragmentation is.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

blackdat78
Program Manager Harvey Nash
Vietnam Vietnam
No Biography provided

Comments and Discussions

 
GeneralThe code has some critical flaws Pinmemberjurujen8-Mar-06 2:43 

An object pool is a pretty common thing, in STL its known as an allocator.
 
But looking at your code im not convinced your original problem was caused by memory fragmentation.
 
You appear to increase the size of your object pool every time you run out of objects in getNewObj() with a call to increaseObjMem().
 
I can see two major flaws with the increaseObjMem() function. The first is that every time you allocate a new larger array you delete the old one, and therfore all of the objects that you previously allocated !
 
if (arrOldObjs != NULL)
delete[] arrOldObjs;
 
Allowing size in the new array for the old objects to be returned is not a solution to this problem, since youve still deleted them. And you can't fix this by copying the old objects either, since they are outside your control and will always reference the memory that used to be part of arrOldObjs[]. What you need is a list of arrays, or buckets and you have to track them all. Only when a bucket is completely empty is it safe to delete.
 
The second problem is that you claim that this is a solution to some memory fragmentation situation but in increaseObjMem() you are calling operator new for every object in your s_arrAllObjs[] pool. Aside from being completely pointless, since all your doing is then asigning its value to a location in the array, your also just slowing your program down and causing lots of memory allocations (which is what you wanted to avoid in the first place).
 
If what you want to do is initialise the array with your objects, then considder using placement new. You can call placement new as required in getNewObj() saving you some performance and a lot of memory allocations, which is after all the entire point of having a pool.
 
But basically, your code at the momement does not work, and causes far greater problems than the imaginary one you were trying to fix.
 

 

GeneralRe: The code has some critical flaws PinmemberVu Thuy30-Aug-07 17:14 
GeneralDon't need this lib if you use MFC Pinmembersonnhq28-Feb-06 5:34 
QuestionPossbile problem? Pinmemberprcarp24-Feb-06 5:20 
AnswerRe: Possbile problem? Pinmemberblackdat7826-Feb-06 18:00 
GeneralFragmentation when freeing in reverse order PinmemberMartin Richter1-Sep-05 3:46 
Generala question!!!! Pinmemberjpopop17-Aug-05 0:33 
GeneralRe: a question!!!! Pinmemberblackdat7817-Aug-05 0:41 
GeneralLook at Boost Pool Library PinmemberProf. Nimnul8-Aug-05 5:36 
GeneralClean up the original code and no leak Pinmembercrash10133-Aug-05 11:56 
GeneralRe: Clean up the original code and no leak Pinmemberblackdat783-Aug-05 17:43 
GeneralInteresting article PinsussAnonymous2-Aug-05 8:44 
GeneralRe: Interesting article Pinmemberblackdat782-Aug-05 17:25 
GeneralRe: Interesting article Pinmembermelwyn3-Aug-05 1:20 
GeneralThx, Very Useful PinmemberHing30-Jul-05 22:25 
GeneralConsider using LFH instead. Pinmemberyafan29-Jul-05 4:33 
GeneralRe: Consider using LFH instead. PinmemberBlake Miller29-Jul-05 5:24 
GeneralRe: Consider using LFH instead. PinsussAnonymous3-Aug-05 2:50 
GeneralRe: Consider using LFH instead. Pinmembererushton15-Aug-05 4:27 
QuestionDoes this actually work for CStrings? PinmemberLuuk Weltevreden28-Jul-05 23:09 
AnswerRe: Does this actually work for CStrings? Pinmemberslim29-Jul-05 2:26 
AnswerRe: Does this actually work for CStrings? Pinmemberarmentage29-Jul-05 9:17 
AnswerRe: Does this actually work for CStrings? Pinmemberblackdat7831-Jul-05 20:58 
GeneralVery interesting PinmemberStlan28-Jul-05 22:07 
GeneralRe: Very interesting Pinmembergnk31-Jul-05 15:01 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.141220.1 | Last Updated 23 Feb 2006
Article Copyright 2005 by blackdat78
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid