Click here to Skip to main content
15,435,841 members
Articles / Programming Languages / XML
Posted 23 Oct 2010

Tagged as


14 bookmarked

Xml Serialization on large xml files

Rate me:
Please Sign up or sign in to vote.
4.96/5 (14 votes)
23 Oct 2010CPOL2 min read
As well all know, xml serialization is a great help in many applications. But it always loads the entire document into memory. For most applications this is not an issue, but what do you do when you have a 100MB xml file (to those whose eyebrows are raising at the thought...yes, they CAN get that big LOL) or perhaps your file is not that insanely massive, but the thought of 5000 objects still makes you seek alternate forms of storage. Well, I'm here to tell you you can still use Xml serialization and not load everything into memory. Sound too good to be true? Read on...

Say for example you have an Xml file that has a bunch of configuration options for your app, etc, but there is one node in there with many hundreds or even thousands of child nodes. In your app, you want to iterate over all of those nodes (deserialized into objects of course). I guess you could use Linq to Xml, but I have found that also get pretty slow with larger files. Here's a better way:

A combination of XmlTextReader and XmlSerializer - really beautiful!

1. Load everything EXCEPT that particular node into memory by using an [XmlIgnore] attribute on the property (or turn it into a method)

2. In your class file, change that property to return an IEnumerator<t>; where T is the class representing your troublesome xml node.

3. In your new property (or method), use an XmlTextReader (for speed) to go through the xml file and whenever you hit a node with the specified name, deserialize it! Then return it with the yield command and voila! There you have it! See code example below to make things clearer (Dunno about all of you, but personally I generally skip over most of an article and go straight to the code; I find it speaks my language just fine) :-D

Here you go:

public IEnumerator<thingy> Thingies
      string xml = string.Empty;
      using (FileStream fs = new FileStream(FileName, FileMode.Open, FileAccess.Read))
         using (StreamReader sr = new StreamReader(fs))
            xml = sr.ReadToEnd();

      using (StringReader stringReader = new StringReader(xml))
         using (XmlTextReader reader = new XmlTextReader(stringReader))
            reader.WhitespaceHandling = WhitespaceHandling.None;
            bool keepGoing = reader.Read();

            while (keepGoing)
               if (reader.IsStartElement())
                  if (reader.Name == "Thingy")
                     yield return Thingy.Parse(reader.ReadOuterXml());
                  else { keepGoing = reader.Read(); }
               else { keepGoing = reader.Read(); }

and if you're using the MBG Extensions Library then "Thingy.Parse" can be as simple as:
public static Thingy Parse(string xml)
   return xml.XmlDeserialize<thingy>();

Go on, give me a 5/5... you know you want to!!! :-D


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Written By
Software Developer (Senior) Freelancer
Australia Australia
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

QuestionExcuse me, but... Pin
Gary R. Wheeler30-Apr-15 4:03
MemberGary R. Wheeler30-Apr-15 4:03 
QuestionMy 5 (but how about SAX parser?) Pin
Andreas Gieriet3-Jun-12 19:41
professionalAndreas Gieriet3-Jun-12 19:41 
GeneralReason for my vote of 5 Cool and handy helper, and a to-the-... Pin
Eddy Vluggen2-Nov-11 6:27
professionalEddy Vluggen2-Nov-11 6:27 
GeneralReason for my vote of 5 Thanks, useful information )) Pin
maxtasha23-Oct-11 14:27
Membermaxtasha23-Oct-11 14:27 
GeneralReason for my vote of 5 Good one...have a 5 Pin
thatraja29-Oct-10 21:07
professionalthatraja29-Oct-10 21:07 
GeneralReason for my vote of 5 good catch Pin
JalalAldeen25-Oct-10 18:27
MemberJalalAldeen25-Oct-10 18:27 
GeneralReason for my vote of 5 Timely assistance on my project exce... Pin
ely_bob25-Oct-10 14:21
professionalely_bob25-Oct-10 14:21 
GeneralReason for my vote of 4 Thanks for Sharing Gordon Matt! Pin
Sivaraman Dhamodharan24-Oct-10 16:24
MemberSivaraman Dhamodharan24-Oct-10 16:24 
GeneralReason for my vote of 5 It's nesassary for my works Pin
Member 729780224-Oct-10 0:23
MemberMember 729780224-Oct-10 0:23 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.