|
Do you NEED to hold all of the data in memory concurrently?
"One man's wage rise is another man's price increase." - Harold Wilson
"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons
"You can easily judge the character of a man by how he treats those who can do nothing for him." - James D. Miles
|
|
|
|
|
yes. the data read from the file to be stored internally.
|
|
|
|
|
I understand that much from your initial post. But does it all need to me in memory concurrently? It's highly unlikely that it does, but only you can be the ultimate judge of that.
What exactly are you doing to/with the data? Maybe an on-disk solution can be suggested.
"One man's wage rise is another man's price increase." - Harold Wilson
"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons
"You can easily judge the character of a man by how he treats those who can do nothing for him." - James D. Miles
|
|
|
|
|
Thank you for the suggestion. I could have tried that if all data were not required and could have read only the required data from the file and store it in memory. But here the need is, after loading of file, it shall take inputs from user either to process all the data or required data, if its all data then it has to have all the data stored in memory.
|
|
|
|
|
divya03 wrote: I tried it in 64-bit 8GB RAM system but still the same exception is thrown
Did you rebuild for a 64 bit target?
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment
"Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst
"I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
|
|
|
|
|
Ya I've rebuild in 64 bit target and run the program in 64 bit system.
|
|
|
|
|
Perhaps there is something useful in this thread[^]. Do you have an estimate of the amount of memory needed?
"the debugger doesn't tell me anything because this code compiles just fine" - random QA comment
"Facebook is where you tell lies to your friends. Twitter is where you tell the truth to strangers." - chriselst
"I don't drink any more... then again, I don't drink any less." - Mike Mullikins uncle
|
|
|
|
|
118Mb isn't all that big. You could easily hold it in memory. The map is costing you the size of the class plus about four pointers worth of storage for each entry. Perhaps your class stores the data very inefficiently. Perhaps you have a bug on reading in the data, like not recognizing end of file and adding endless entries until you run out of memory. Perhaps you have a memory leak.
- How many rows are in the file? You can easily build a function to count rows using the current function that inputs data, but without saving the data.
- How many rows are read in at the time the out-of-memory exception occurs? This and the last question might tell you if you had a bug inputting the data.
- Are you using std::string to hold any of the row data, because memory costs can really add up when using std::string.
Maybe your representation is too expensive for the memory you have. You could approach this problem by reading in the file as a block, and building a data structure that puts pointers into the file block for each line, and then parse out a single line when you need one.
|
|
|
|
|
Total number of rows are around 7 lakhs. The out of memory is occurring after around 4lakhs rows are read and when the memory in task manager shows around 700000k by the application. Yes std:string is used as a member variable of class object.
|
|
|
|
|
On average, std::string allocates enough memory to hold 1.5x the size of the actual string. (This is a rule of thumb), plus 2 pointers of overhead. This could be your problem, depending how many strings you have per row. Depending how you've written your data structure, it's quite likely that you have really exhausted memory. You need to rethink the problem so to solve it using less memory.
|
|
|
|
|
Are you sure the memory is actually the issue and you aren't hitting the class object limit. From memory the allocation just starts failing when you hit the 10,000 limit. Adding more memory doesn't help because it isn't the problem in the first place, you just can't allocate any more objects. Easy to check count objects as you create them and see what count you get to.
In vino veritas
modified 13-Aug-17 1:04am.
|
|
|
|
|
Class object limit? What's that?
|
|
|
|
|
Have you tried catching std::bad_alloc exception?
|
|
|
|
|
what is the value of y after the following statements?
float y;
y = 17 / 5;
|
|
|
|
|
Probably 3, because you're dividing two integers and then storing the resulting integer in a float.
But why don't you run it for yourself and find out?
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
As Richard said, the expression only contains integers and so you should get the result of an integer division (3).
However, if at least one of the operands / variables are of float type, all integers will be automatically converted to float type.
Say, for example - y = (float)17 / 5;
Having said this, the answer may be slightly different from the expected 3.4 because of how floating point numbers are represented.
Please check this - Floating point inaccuracy examples - Stack Overflow[^]
«_Superman_»
I love work. It gives me something to do between weekends.
Microsoft MVP (Visual C++) (October 2009 - September 2013) Polymorphism in C
|
|
|
|
|
I have USB drive inserted in computer. My application should be notified when user click in eject button from system tray.
Is there any way to get notification?
|
|
|
|
|
|
Very good reply! That is a little known and obscure part of the win API.
|
|
|
|
|
|
Randor wrote: maybe Jochen Arndt[^] is a bot?
By his profile picture, he's a boot (or possibly a pair of them) 
If you have an important point to make, don't try to be subtle or clever. Use a pile driver. Hit the point once. Then come back and hit it again. Then hit it a third time - a tremendous whack.
--Winston Churchill
|
|
|
|
|
It is still a little known and obscure part of the API.
|
|
|
|
|
Thanks Jochen,
WM_DEVICECHANGE message is to handle device removal and insertion event. But my requirement is something different.
I want to handle a notification when user click on "eject" option from system tray. This event should occur before WM_DEVICECHANGE I think.
Please help me to do so.
|
|
|
|
|
It can be also received by the WM_DEVICECHANGE handler as DBT_DEVICEQUERYREMOVE, DBT_DEVICEREMOVEPENDING, or DBT_CUSTOMEVENT.
This requires calling RegisterDeviceNotification with appropriate settings. But that will not support volumes (DBT_DEVTYP_VOLUME) because arrival and removal evens are send by default. The only solution I know is opening a device handle for the USB drive and using that to register. Than you will get informed about the eject request and can reject the request or accept it after closing the device handle. See Device Events (Windows)[^].
|
|
|
|
|
Thanks Jochen,
Your approach worked well with MFC application but same approach not working with windows service. Any clue?
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
