I am trying to build a log analyzer which reads each log file one by one (may extend to more than 500 text files of 2 MB each).
Every time I open a log file, parse it, separate all the logs according to each thread. close the CStdioFile pointer and then delete it. ( So there is only one log file opened at any point of time for reading)
I have a map which has key as threadID (integer) and second as a file pointer. (I create a separate file for each thread, earlier i used to store each log entry in vector whose pointer I stored in second of this map, but now using files because of memory issues)
I have implemented logic to close 100 file pointers(from beginning of map) whenever total open files reach 500. So at any point of time only 500 files are open, (NO THREADING ISSUE)
Everything works fine for reasonable number of threads (about 700-800), but after that CStdioFile is unable to open any new or existing file for no reason, and throws CFileException with
Point at which exception is thrown is not defined, sometimes it may go upto creating 1500 files and sometimes as low as 800.
I am unable to find any particular reason for this.
Thanks in advance..