Click here to Skip to main content
15,032,967 members
Please Sign up or sign in to vote.
5.00/5 (1 vote)
See more:
I am using Visual C (win32 Console Application) for feature extraction program. In my project I am extracting features from the trained images. Features aren in form of structures. Strcuture has 4 DOUBLE data items and 1 INT array of size 128. So this means one feature from the image. On average one image may generate from 500-1000 features.

I am using dynamic memory allocation to store the structures for every trained image. I am using 800 images for training. This means huge amount of memory will be consumed to keep these features iN RAM. Now problem is that after training of 740 images I am getting error of memory leak i.e. insufficient memory to allocate. I have tried every possible way to avoid it but still cannot solve the issue. So please tell me how i can overcome this issue. I will be really grateful.
Posted

I recommend you map the file directly into memory, and just type cast the pointer to a pointer to your struct. Then it's a small deal indexing this array.

This is probably as fast as it gets, and you don't have to allocate/deallocate any memory.

Example of the concept:
struct Feature
{
    double d1, d2, d3, d4;
    int arr[128];
};

HANDLE hFile = ::CreateFile(path, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, 0);

size_t nFileSize = ::GetFileSize(hFile, NULL);
HANDLE hMem = ::CreateFileMapping(hFile, NULL, PAGE_READONLY, 0, 0, NULL);

Feature *pFeatures = (Feature*)::MapViewOfFile(hMem, FILE_MAP_READ, 0, 0, 0);

size_t featureCount = nFileSize / sizeof(Feature);

for (size_t f = 0; f < featureCount; ++f)
{
    Feature *pFeature = &pFeatures[f];
    // Use pFeature here
    // As long as you have the file mapping open, you can use the pointers.
}

Error checking, and closing of the file mapping, omitted.

You might need to align the members in the Feature struct depending on your data. But not as it is right now.

Edit: Minor code changes
   
v4
Comments
Aescleal 8-Sep-10 11:00am
   
If he can't allocate memory then he's probably running out of enough address space to allocate a large enough block of memory in. If that's the case then using a memory mapped file isn't going to help that much.

If he's really running out of memory then it's probably better to increase the size of the page file.
Niklas L 9-Sep-10 2:57am
   
That can also be solved by mapping smaller regions of the file into memory. You only need to map those you are actually using at any one time + a few more for performance. Wrap it up in a class, and you have full control of the bookkeeping.
I am getting error of memory leak i.e. insufficient memory to allocate.

They are two different thing:
A "memory leak" is when you loose the reachability of a memory block you allocate, so that you cannot free it when it is anymore need it.
Large amount of memory leaks can cause insufficient memory condition.
Sometimes -viceversa- an insufficient memory condition may throw some excetion/cause premature function exiting, so that some memory is lost somewhere causing leaks.

In general both problem are symptoms of "bad design", and are rarely solvable as "problem to be fixed".
You've have to better design the way memory allocation/release is managed, sine is highly probable that "real" numbers are much and much more than the ones you talked about.
   
A memory leak is when you don't free memory at some point after you allocate it.

You've allocated ~800 structures at about 0.5k a structure (give or take some shouting) so that should only be about 400K. These days this isn't a large amount of memory (unless you're using some sort of embedded system in which case dynamic memory allocation is probably a no-no). I'd have expected you to be able to create ~2000 times that and get away with it.

So which way are you allocating memory for this lot?
   
v2
Comments
CPallini 8-Sep-10 3:43am
   
Actually he allocated ~800 images, every one generating (at least) 500 features of about 0.5 kB, i.e. ~200 MB.
Aescleal 8-Sep-10 4:53am
   
Good point - he's still well within the limits of what most processes can hold though.
Aescleal 8-Sep-10 4:55am
   
Although thinking about it if he's storing a huge image in memory at the same time he might be in trouble on a 32 bit OS.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)




CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900