Problem in Memory Allocation in C++ for feature extraction

Question

5.00/5 (1 vote)

See more:

I am using Visual C (win32 Console Application) for feature extraction program. In my project I am extracting features from the trained images. Features aren in form of structures. Strcuture has 4 DOUBLE data items and 1 INT array of size 128. So this means one feature from the image. On average one image may generate from 500-1000 features.

I am using dynamic memory allocation to store the structures for every trained image. I am using 800 images for training. This means huge amount of memory will be consumed to keep these features iN RAM. Now problem is that after training of 740 images I am getting error of memory leak i.e. insufficient memory to allocate. I have tried every possible way to avoid it but still cannot solve the issue. So please tell me how i can overcome this issue. I will be really grateful.

Posted 7-Sep-10 15:50pm

Nab Younus

Add a Solution

3 solutions

Solution 3

I recommend you map the file directly into memory, and just type cast the pointer to a pointer to your struct. Then it's a small deal indexing this array.

This is probably as fast as it gets, and you don't have to allocate/deallocate any memory.

Example of the concept:

struct Feature
{
    double d1, d2, d3, d4;
    int arr[128];
};

HANDLE hFile = ::CreateFile(path, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, 0);

size_t nFileSize = ::GetFileSize(hFile, NULL);
HANDLE hMem = ::CreateFileMapping(hFile, NULL, PAGE_READONLY, 0, 0, NULL);

Feature *pFeatures = (Feature*)::MapViewOfFile(hMem, FILE_MAP_READ, 0, 0, 0);

size_t featureCount = nFileSize / sizeof(Feature);

for (size_t f = 0; f < featureCount; ++f)
{
    Feature *pFeature = &pFeatures[f];
    // Use pFeature here
    // As long as you have the file mapping open, you can use the pointers.
}

Error checking, and closing of the file mapping, omitted.

You might need to align the members in the Feature struct depending on your data. But not as it is right now.

Edit: Minor code changes

Posted 8-Sep-10 1:11am

Niklas L

Updated 8-Sep-10 1:15am

v4

Comments

Aescleal 8-Sep-10 11:00am

If he can't allocate memory then he's probably running out of enough address space to allocate a large enough block of memory in. If that's the case then using a memory mapped file isn't going to help that much.

If he's really running out of memory then it's probably better to increase the size of the page file.

Niklas L 9-Sep-10 2:57am

That can also be solved by mapping smaller regions of the file into memory. You only need to map those you are actually using at any one time + a few more for performance. Wrap it up in a class, and you have full control of the bookkeeping.

Solution 2

I am getting error of memory leak i.e. insufficient memory to allocate.

They are two different thing:
A "memory leak" is when you loose the reachability of a memory block you allocate, so that you cannot free it when it is anymore need it.
Large amount of memory leaks can cause insufficient memory condition.
Sometimes -viceversa- an insufficient memory condition may throw some excetion/cause premature function exiting, so that some memory is lost somewhere causing leaks.

In general both problem are symptoms of "bad design", and are rarely solvable as "problem to be fixed".
You've have to better design the way memory allocation/release is managed, sine is highly probable that "real" numbers are much and much more than the ones you talked about.

Posted 7-Sep-10 22:06pm

Emilio Garavaglia

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Aescleal · Accepted Answer · 2010-09-07T21:33:00

Solution 1

A memory leak is when you don't free memory at some point after you allocate it.

You've allocated ~800 structures at about 0.5k a structure (give or take some shouting) so that should only be about 400K. These days this isn't a large amount of memory (unless you're using some sort of embedded system in which case dynamic memory allocation is probably a no-no). I'd have expected you to be able to create ~2000 times that and get away with it.

So which way are you allocating memory for this lot?

Posted 7-Sep-10 21:33pm

Aescleal

Updated 7-Sep-10 21:35pm

v2

Comments

CPallini 8-Sep-10 3:43am

Actually he allocated ~800 images, every one generating (at least) 500 features of about 0.5 kB, i.e. ~200 MB.

Aescleal 8-Sep-10 4:53am

Good point - he's still well within the limits of what most processes can hold though.

Aescleal 8-Sep-10 4:55am

Although thinking about it if he's storing a huge image in memory at the same time he might be in trouble on a 32 bit OS.