Click here to Skip to main content
15,867,835 members
Articles / Programming Languages / C#

How to Open Large XML files without Loading the XML Files?

Rate me:
Please Sign up or sign in to vote.
4.83/5 (11 votes)
12 Feb 2011CPOL1 min read 77.4K   17   3
How to Open Large XML files without Loading the XML Files?

Working with XML files in memory is always a performance issue. It becomes more important to look into the processing of XML files which are heavy in size (let's say more than 3 GB). So questions comes in mind that how to process such heavy XML files.

When we think of working with any XML file, we normally think of using:

  • XMLDocument
  • DataSet.ReadXml()
  • XPathDocument

When we use the above options, we are loading the files into the system memory.

The problem is that, if the size of the XML file is for e.g. 5 GB to 7 GB, we have to load the complete file in System’s memory. This will cost us systems memory and will throw “System out of Memory Exception”.

The best approach to process such large files is to avoid loading the files into the memory.

Microsoft has provided with XmlTextReader class. XmlTextReader helps us to process the XML file line by line. In this way, we are not loading the complete XML file into the memory but processing the file line by line, node by node.

Here is the code snippet that shows an example of how to use XMLTextReader class:

C#
XmlTextReader myTextReader = new XmlTextReader(filename);
myTextReader.WhitespaceHandling = WhitespaceHandling.None;
while (myTextReader.Read())
{
    if (myTextReader.NodeType == XmlNodeType.Element &&
        myTextReader.LocalName == "Reward" &&
        myTextReader.IsStartElement() == true)
        {
            ProcessRewardNode(myTextReader);
                myTextReader.Skip();
    }
}

Here is the method implementation of ProcessRewardNode:

C#
private void ProcessRewardNode(XmlTextReader RewardReader)
{
    XmlDocument RewardXmlDoc = new XmlDocument();
    RewardXmlDoc.LoadXml(RewardReader.ReadOuterXml());
    // we can use xpath as below
    myID = RewardXmlDoc.SelectSingleNode("Reward/myID").InnerText;
}

Here code itself tells you lots of things, so I am not discussing it more here. You can look into MSDN of XMLTextReader for more information.

Hope this will help !!!

Jay Ganesh


License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Technical Lead
India India
I write software using Microsoft web technologies since 2008. I have successfully delivered software products for Fortune 500 companies and startups.

Microsoft Certified Technologies Specialist - Web Applications Development with Microsoft .NET Framework 4.

Awarded as Microsoft Community Contributor of the year 2011.

Received several awards at various forums and my various articles got listed as "Article of the day" at ASP.NET Microsoft Official Website https://www.asp.net/

Visit My Blog:
https://ramanisandeep.wordpress.com/


Area of Expertise:
C#, ASP.NET, Web Services, WCF, ASP.NET MVC, SQL Server, WEB API, AngularJS, jQuery

Comments and Discussions

 
Generalregarding myID = RewardXmlDoc.SelectSingleNode("Reward/myID").InnerText; Pin
Angsuman Chakraborty5-Mar-11 21:15
Angsuman Chakraborty5-Mar-11 21:15 
GeneralMy vote of 5 Pin
yarp13-Feb-11 18:28
yarp13-Feb-11 18:28 
GeneralMy vote of 5 Pin
Tarun.K.S13-Feb-11 5:42
Tarun.K.S13-Feb-11 5:42 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.