Click here to Skip to main content
13,147,183 members (28,744 online)
Click here to Skip to main content
Add your own
alternative version

Tagged as


14 bookmarked
Posted 23 Oct 2010

Xml Serialization on large xml files

, 23 Oct 2010
Rate this:
Please Sign up or sign in to vote.
As well all know, xml serialization is a great help in many applications. But it always loads the entire document into memory. For most applications this is not an issue, but what do you do when you have a 100MB xml file (to those whose eyebrows are raising at the thought...yes, they CAN get that big LOL) or perhaps your file is not that insanely massive, but the thought of 5000 objects still makes you seek alternate forms of storage. Well, I'm here to tell you you can still use Xml serialization and not load everything into memory. Sound too good to be true? Read on...

Say for example you have an Xml file that has a bunch of configuration options for your app, etc, but there is one node in there with many hundreds or even thousands of child nodes. In your app, you want to iterate over all of those nodes (deserialized into objects of course). I guess you could use Linq to Xml, but I have found that also get pretty slow with larger files. Here's a better way:

A combination of XmlTextReader and XmlSerializer - really beautiful!

1. Load everything EXCEPT that particular node into memory by using an [XmlIgnore] attribute on the property (or turn it into a method)

2. In your class file, change that property to return an IEnumerator<t>; where T is the class representing your troublesome xml node.

3. In your new property (or method), use an XmlTextReader (for speed) to go through the xml file and whenever you hit a node with the specified name, deserialize it! Then return it with the yield command and voila! There you have it! See code example below to make things clearer (Dunno about all of you, but personally I generally skip over most of an article and go straight to the code; I find it speaks my language just fine) :-D

Here you go:

public IEnumerator<thingy> Thingies
      string xml = string.Empty;
      using (FileStream fs = new FileStream(FileName, FileMode.Open, FileAccess.Read))
         using (StreamReader sr = new StreamReader(fs))
            xml = sr.ReadToEnd();
      using (StringReader stringReader = new StringReader(xml))
         using (XmlTextReader reader = new XmlTextReader(stringReader))
            reader.WhitespaceHandling = WhitespaceHandling.None;
            bool keepGoing = reader.Read();
            while (keepGoing)
               if (reader.IsStartElement())
                  if (reader.Name == "Thingy")
                     yield return Thingy.Parse(reader.ReadOuterXml());
                  else { keepGoing = reader.Read(); }
               else { keepGoing = reader.Read(); }

and if you're using the MBG Extensions Library then "Thingy.Parse" can be as simple as:
public static Thingy Parse(string xml)
   return xml.XmlDeserialize<thingy>();

Go on, give me a 5/5... you know you want to!!! :-D


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Software Developer (Senior) Freelancer
Australia Australia
No Biography provided

You may also be interested in...

Comments and Discussions

QuestionMy 5 (but how about SAX parser?) Pin
Andreas Gieriet3-Jun-12 19:41
memberAndreas Gieriet3-Jun-12 19:41 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.170915.1 | Last Updated 24 Oct 2010
Article Copyright 2010 by vnmatt
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid