Parsing/Loading/Searching xml document of size ~ 1GB

• Download source files - 56.5 Kb
• Download demo project - 446 Kb

Introduction

The TechFestXmlSolution is a tool which help the developer in parsing large xml documents this application is specific for perticular xml file and is taken from Project Gutenberg(http://www.gutenberg.org/) maintains a list of books in RDF format The application is used for searching the xml file for perticular bookid,getting books by index and searching book by text

Perforamance and scalability issue:

Processing large xml document using Dom object causes a high CPU,Memory and Bandwidth utiliation
Appropriate design decisions can help you address many XML-related performance issues early by chossing appropriate XML class for the job considering combining xmlReader and xmlDocument and on the xmlReader use moveToContent and Skip methods. We can read upto 2 GB using xmlReader as it only load 4Kb buffers into memory.

Using the Code

      In this application we have used xmlReader to go through each line and xmlDocument object to load the part of the xml . xmlReader.moveToContent() method Checks whether the current node is a content (non-white space text, CDATA, Element, EndElement, EntityReference, or EndEntity) node. If the node is not a content node, the reader skips ahead to the next content node or end of file. It skips over nodes of the following type: ProcessingInstruction, DocumentType, Comment, Whitespace, or SignificantWhitespace    and xmlReader.Skip() method to skip children of the current node which we do not have to search . See the snap shot of the code

while (!_xReader.EOF)
{  // check Node type,node name ,match attribute which is id to search, RDF is root element
   if((_xReader.MoveToContent() == XmlNodeType.Element && _xReader.Name == "pgterms:etext" 
         && _xReader.GetAttribute(0) == name) || _xReader.Name == "rdf:RDF")
    { // if the node name is rdf the it is root element don't skip it and continue to read            
     if (_xReader.Name == "rdf:RDF")
      {
        Console.WriteLine(" Before finding the book  CPU usage  ->"+sampleCounter.CpuUsage);
       _xReader.Read();
      }
     else
      {   // when the node containing Id is found  create the document
          // object to load the node only here we are loading the node
          // in document not whole file for getting the data of node faster
       Console.WriteLine(" After finding the book CPU usage  ->" + sampleCounter.CpuUsage);
       doc = new XmlDocument();
          // to get node in memory 
       XmlNode xnode = doc.ReadNode(_xReader);
          // check if element contains any attribute 
       if (xnode.Attributes.Count > 0)
        {  // call the Initialise method of class Book which intializes whole variables
           book1.Initialise(xnode);
        }
          // Print the whole book description after initialising 
       break;
      }
   }
  else
   {     // skip the whole node as it is not of use
      _xReader.Skip();
      _xReader.MoveToContent();
   }

Similarly the application contains 2 more function for getting the books from index and searching the book by title/subject/publisher etc  while searching the books at the back this function are writing the output in the text file the path of the text file is given in the App.Config please

       /// Function getBooks takes start and end index as argument to find that number of books
       /// and retunrns the books in list of type Book 
    public override List getBooks(long _startIndex, long _lastIndex)
    {
        // create the List object of capacity number of books needed
      List bookList = new List(Convert.ToInt32(_lastIndex - _startIndex + 1));
       // local variable to count number of book found
      int _index = 0;
      sampleCounter.StartTime = DateTime.Now;
      XmlReader _xReader = openXmlFile(_fileName);
      try
      {   // loop till end of the file 
        while (!_xReader.EOF)
         {  // check the conditions that untill end file 
          if ((_xReader.MoveToContent() == XmlNodeType.Element && _xReader.Name == "pgterms:etext" 
                && _index > _startIndex - 2 && _index < _lastIndex) || _xReader.Name == "rdf:RDF")
               {
                 if (_xReader.Name == "rdf:RDF")
                  {
                    _xReader.Read();
                  }
                 else
                  {
                     // index is greater than start index and less than lastIndex
                     // get whole node in the memory so that searching become easy and productive
                     doc = new XmlDocument();
                     XmlNode xnode = doc.ReadNode(_xReader);
                     // create the instance of the book as an container 
                     Book book1 = new Book();
                     if (xnode.Attributes.Count > 0)
                      { // Intialise and add to the list 
                           book1.Initialise(xnode);
                           bookList.Add(book1);
                      }
                    // increment as book is found
                   _index++;
                  }
                 } // check whether the found books are equal to last index or that 
             else if (_index == _lastIndex)
              {  // last index is reached break 
                    break;
               }
             else  // check until start index is not reach 
              {
               if(_xReader.MoveToContent() == XmlNodeType.Element && _xReader.Name == "pgterms:etext")
                {
                    _index++;
                    _xReader.Skip();
                    _xReader.MoveToContent();
                }// skip unwanted nodes to make efficent searching
               else
                {
                   _xReader.Skip();
                   _xReader.MoveToContent();
                }
              }
         }
       }catch (Exception ex){// Error    }
       finally{ _xReader.Close(); }
        	// Check wether list conatins any book or not
      if (bookList.Count > 0)
       {
          Console.WriteLine("Total number of books are " + bookList.Count);
          Console.WriteLine(" CPU usage is " +sampleCounter.CpuUsage);
          Console.WriteLine("Time taken " + sampleCounter.TimeTaken);
            // return the whole list of Books 
          return bookList;
        }
        else 
          // out of range returns null
          return null;
        }

Points to Remember

Please see the schema of the xml file for such type  which only the application can be used
If you want to use this function in your application make certain changes in the GutenbergBookManager.cs class
Make sure that the path of the xml file which you have to parse must be given in the App.config for more details read the documents in zip file