Click here to Skip to main content
Rate this: bad
good
Please Sign up or sign in to vote.
See more: MFC VC++
I am trying to build a log analyzer which reads each log file one by one (may extend to more than 500 text files of 2 MB each).
Every time I open a log file, parse it, separate all the logs according to each thread. close the CStdioFile pointer and then delete it. ( So there is only one log file opened at any point of time for reading)
 
Parsing:
I have a map which has key as threadID (integer) and second as a file pointer. (I create a separate file for each thread, earlier i used to store each log entry in vector whose pointer I stored in second of this map, but now using files because of memory issues)
 
I have implemented logic to close 100 file pointers(from beginning of map) whenever total open files reach 500. So at any point of time only 500 files are open, (NO THREADING ISSUE)
 
Everything works fine for reasonable number of threads (about 700-800), but after that CStdioFile is unable to open any new or existing file for no reason, and throws CFileException with
m_cause =0(none).
 
Point at which exception is thrown is not defined, sometimes it may go upto creating 1500 files and sometimes as low as 800.

I am unable to find any particular reason for this.
 
Thanks in advance..
Posted 13-Mar-13 11:17am
Comments
nv3 at 13-Mar-13 17:06pm
   
Sorry, but I found it hard to follow the story. In the first paragraph you say that there is only one log file open at a time. Then you describe that you close a hundred files whenever the number of open files exceeds 500. You have one file per thread and 700-800 threads, but only 500 files open. Perhaps you can explain all that a little more detailed.
 
Regarding the exception: What have you done to analyze it? Can you reproduce it in a debugger session? (That would allow you to look around at variable values and probably see the problem relatively quickly.) Have you tried to catch the exception and do some debug prints? Just ideas about what you can do to solve this case.
 
Just an idea: When the problem occurs when opening a file after some 800 or 1500 open operations, I would suspect that your file closing is not working properly and you are running into the limit of maximum open files. But this is just a guess.
Andrew Cherednik at 13-Mar-13 21:05pm
   
I suggest you monitor your program using a tool as easy accessible as task manager. In the task manager use View->Select Columns. There, select Thread Count, Handle Count, memory related columns, GDI Handles (I doubt they are involved). View the numbers when error happens. I bet one of them will be unusually high

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
0 OriginalGriff 277
1 Maciej Los 220
2 DamithSL 190
3 Raul Iloc 185
4 Richard MacCutchan 175
0 OriginalGriff 5,130
1 DamithSL 4,197
2 Maciej Los 3,670
3 Kornfeld Eliyahu Peter 3,470
4 Sergey Alexandrovich Kryukov 2,821


Advertise | Privacy | Mobile
Web03 | 2.8.141216.1 | Last Updated 13 Mar 2013
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100