Click here to Skip to main content
Click here to Skip to main content
Go to top

Handling Large Data Sets in Silverlight using WCF and Customized DataGrids

, 3 Jul 2010
Rate this:
Please Sign up or sign in to vote.
Display, edit and save large data sets in Silverlight using WCF based services and customized DataGrids

Introduction

This article gives you a brief introduction in advanced techniques for up- and downloading large datasets and editing them. Therefore only standard libraries delivered with Silverlight 3 are used. The communication is based on WCF services and the UI only uses datagrids, buttons and textboxes. The interesting part of this article is that it shows you how to overcome some common obstacle that you easily can bump into by just using extension classes, reflection and some simple but powerful tricks.

Background

When I started using Silverlight, I took a brief look at some examples and built the first sample applications very fast. I had some small problems with data binding and XAML but after understanding the basic concepts, everything seemed to be easy. I found many very good examples covering the basic concepts which gave me a good start and made things look easy. Then I had to implement the first real application and things became problematic. What I wanted to implement seemed to not be very difficult - when you would implement it in WinForms - but with Silverlight, I soon found out that it would not work the way you would do it with WinForms/MFC. Here is what the application should do:

  • Handle large amount of data: up- and download over 50,000 rows of data and more than 10 MB of data
  • Display the data in a datagrid fast (high performance)
  • Add data without using a pop-up dialog - just always have an empty line at the end
  • Sort data
  • Filter data

All this sounds simple but it is not. The first problem is that uploading more than 8 KB of data using WCF services does not work out of the box. The second problem is that adding an empty line to a datagrid is easy, but in combination with sorting the empty line is sorted as well and is not always at the end: if you change sort direction, it becomes the first row. And finally performance can become a problem. Sorting 50,000 rows takes several seconds when using an unmodified datagrid and this is definitely not satisfying, especially if you compare it to WinForms.

Overview of this Solution

Basically this solution consists of two parts:

  • The WCF service for handling large data sets
  • The extensions to the datagrid for displaying, editing, sorting and filtering the data

First we will take a look at the WCF service so that we have some data to play with. Then we will take a look at the datagrid. This demo uses Silverlight 3.0, .NET 3.5 and was written with Visual Studio 2008. It should run by just opening the solution and pressing F5. I will not go into detail on how to deploy it (client and server on different machines) or go too much into detail on how to define services, XAML files, etc. using Visual Studio 2008. There already exist a lot of very good examples - also here on CodeProject - so I think I could not make it better or add something to these examples.

The WCF Service

The WCF service is implemented server side and part of the "MappingDataEditor.Web" project. Loading large amounts of data works (almost) out of the box. In this example, the content of "test_data.csv" (located in the subdirectoy "TestData") is loaded 50 times. The file has 1,000 rows so 50,000 rows are loaded resulting in ca. 11 MB of data sent from the server to the client. In the "web.config", make sure to set "dataContractSerializer" parameter:

<dataContractSerializer maxItemsInObjectGraph="2147483647"/>

Uploading is a little bit more complicated because you cannot upload more than 8 KB with one call. So the data is transferred in blocks. Client side is managed by the UploadState-object. The methods needed on the server side look like this:

[OperationContract]
Guid GetHandle(string pTableame);

[OperationContract]
void AddChunk(Guid pHandle, string pBlock);

[OperationContract]
bool Complete(Guid pHandle);

The data is transferred using the following steps:

  • The data in the table is serialized
  • The UploadState-object is initialized and after calling StartUpload communication is handled completely by the UploadState-object
  • The first block is sent to the server using the WCF call "GetHandle"
  • Then the data is transferred using 8KB blocks with multiple calls to "PushOneBlock"
  • After each block transfer the calling process is informed about the progress (you can register on the "ProgressChanged" event)
  • After transferring the final block, the method "Complete" is called. Now the server side service knows that all data is transferred and can do whatever should be done with the data (in this case, saving data)

The Datagrid

The datagrid uses the SortCollectionView object which implements the interfaces ICollectionView and IList.

public class SortCollectionView<T> : ICollectionView, IList<T>
{
    private ObservableCollection<T> myContent = new ObservableCollection<T>();
    ...

Now sorting and filtering can be implemented by ourselves. This way we can make sure that an empty line for entering new data is always at the end. We just remove the last line, order the data and add the empty line again:

object selectedItem = myDGTable1.SelectedItem;
SortDescription sortDesc = myReturnTableView.SortDescriptions[0];
this.myReturnTableView.Content.RemoveAt(this.myReturnTableView.Content.Count - 1);
if (myChGeneric.IsChecked == true)
{
    myReturnTableView.Content.SortQuick(new GenericComparer<ReturnTableEntry>
	(sortDesc.PropertyName, sortDesc.Direction == ListSortDirection.Ascending, 
	StringComparison.CurrentCulture, myReturnTableView[0]));
}
else
{
    if (sortDesc.Direction == ListSortDirection.Ascending)
    {
        myReturnTableView.Content.SortQuick(new Field1ComparerUp());
    }
    else
    {
        myReturnTableView.Content.SortQuick(new Field1ComparerDown());
    }
}

myDGTable1.SelectedItem = null;
myReturnTableView.Content.Add(new ReturnTableEntry());
myReturnTableView.FireCollectionChanged();
myDGTable1.SelectedItem = selectedItem;

Also we can implement filtering by passing a FilterCallback function and applying it on the data:

public IEnumerator<T> GetEnumerator()
{
    if (myFilter == null)
    {
        return this.myContent.GetEnumerator();
    }
    else
    {
        Collection<T> col = new Collection<t />();
        foreach (T item in this.myContent)
        {
            if (myFilter(item))
            {
                col.Add(item);
            }
        }

        return col.GetEnumerator();
    }
}

Using the Code

Now that we took a brief look at the interesting parts of this example application, here is a listing of the interesting files and what they are used for:

  • MainPage.xaml: UI definition in XAML defining the layout of Tabpage, Canvas, Table, Buttons and Textboxes
  • MainPage.xaml.cs: Code behind file containing most of the UI logic
  • SortCollectionView.cs: Object for sorting and filtering an ObservableCollection
  • ChangedSortDescriptionCollection.cs: Needed to get access to the "CollectionChanged" event
  • Extensions.cs: Generic extension methods for sorting ObservableCollection (try out the Bubblesort and see how slow it is compared to Quicksort)
  • UploadState.cs: Helper class for uploading large amounts of data as a string. Extending this one with a ZIP-compressor (a great implementation can be found here) could speed up the upload.
  • IDataService.cs: Interface of the WCF service
  • DataService.svc.cs: Server side implementation of the WCF service
  • DataServiceHostFactory.cs: "ServiceHostFactory" for the service to prevent problems when you use multiple services
  • Default.aspx: Website hosting the Silverlight application
  • UploadManager.cs: Keeps track of the received block when uploading the data in multiple small blocks of data
  • UploadSession.cs: Implementation of one upload session
  • UploadSessions.cs: Cache for all currently active "UploadSession"s
  • ReturnTableEntry.cs: Simple object used in this example for representing one line in the table

Points of Interest - Performance Considerations

There are some very interesting observations I made while evaluating performance. The sorting is based on a Quicksort algorithm. First I implemented a generic sorter which determines by using reflection which type the ordered column has:

this.myPropertyName = value;
this.myPropInfo = this.myTypeInstance.GetType().GetProperty(this.PropertyName);
if (this.myPropInfo.PropertyType.Equals(typeof(string)))
{
    this.CompareFunction = this.StringComparer;
}
else if (this.myPropInfo.PropertyType.Equals(typeof(int)))
{
    this.CompareFunction = this.IntComparer;
}
else if (...

Depending on the type, it sets the comparer that should be used. When sorting the data, the comparer gets the values using reflection again:

private int StringComparer(T pTypeInstance1, T pTypeInstance2)
{
    string string1 = (string)myPropInfo.GetValue(pTypeInstance1, null);
    string string2 = (string)myPropInfo.GetValue(pTypeInstance2, null);
    ...

On a Core2Duo with 2.7 GHz, it takes more than 8 seconds sorting the 50,000 rows in the table. But if you do not use reflection and pass a comparer which accesses the properties directly, the sorting takes less than 1 second making everything more than 10 times faster. So reflection is great when you want to implement a generic solution but it has a great impact on performance.

The second interesting thing is that the first time you execute some code, it always takes approximately 20% more time than the following calls. Here you can see the impact of the .NET interpreter on the code. You can try this out by taking a look at the lower part of the application. In the textbox, you can see how long the operations last.

Also take a look at the SortCollectionView. Internally it stores the data in an ObservableCollection. This way you can set the data directly as it comes from the WCF service without any conversion. So setting the data and displaying it only takes 0.1 second. If you would copy the data over into another format, it would take a few seconds.

Remarks

This is just an example application and you should not use it in a productive environment before testing it thoroughly. Also this implementation only works with ObservableCollections and Properties. It will not work with DataViews. Anyway I hope you like it, and that it saves you a lot of time. Have fun with it!

History

  • 5th April, 2010: Version 0.1 for Silverlight 3 of the test application "MappingDataEditor"
  • 2nd July, 2010: New version for Silverlight 4.0 and .NET 4.0
    • Removed a bug with the checkbox
    • Added functionality for paged loading
    • Remark: Performance not optimized for large datasets. Please take a look at TPL and parallel loops
    • The access path to the Testdata is hardcoded in the Web.config. Please edit it or extract the project folder directly to "C:\"
    • Pressing the "create"-Button generates a new, large TestData-File with 100000 lines. After this loading data in one Block could fail. Only Paged load works then. Adjust the number of lines that will be created in "DataService.svc.cs".

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

bl@cksheep
Team Leader maxment GmbH
Germany Germany
I wrote my first program more than 30 years ago when I was 8 years old. It was written in Pascal on an Apple ][e and calculated the Greatest Common Divisor. I was so proud when it finally work - from that day on computers where part of my life.
 
After successfully studying computer science I worked in a large cardiologic department as software developer. A few years later I went back to university and my PhD in medical computer science (Dr. sc. inf. biomed.). During this time a lead small team of really smart develeopers and we implemented the market leading cardiologic information system for the German market. In the following years I worked for Siemens but I missed this inspiring, creative athmosphere of a small company. So I changed back to a small start-up.
 
We are specialized on Webdevelopment and Windows technologies (C#, .Net, Silverlight, WPF, WCF, WinForms, Prism etc.). We are also experts in in Databases (ORACLE, MS-SQL-Server, PostgreSQL), medical standards (HL7, DICOM, IHE) and coordination of larger projects (including techniques like Agile Development and SCRUM). So if you are looking for a smart and efficient developer just contact me.
 
I started with C# and .NET in 2003. When Silverlight 2 hit the market I took a look at it and never let go again. Now - after two years and more than 500,000 lines of code - I think that I have a good theoretical and pratical understanding of this technology. So I would like to give back something to the community and share some tricks with you. I hope that it helps somebody - like all the interesting articles I read while learning the secrets of Silverlight, C#, .NET ...

Comments and Discussions

 
QuestionGreat job man!!!!! Pinmembertjebbour1-Jul-11 8:26 
GeneralMy vote of 4 PinmemberVishnu Rana7-Nov-10 3:05 
GeneralSL 4 version and large datasets Pinmemberbl@cksheep2-Jul-10 11:05 
Generalvery large data Pinmemberimehlhorn2-Jul-10 3:43 
GeneralRe: very large data Pinmemberbl@cksheep2-Jul-10 11:08 
I just send an update to the admins which shows the basic idea of paged loading. I think they will update the page soon. You can also send me an email to "marcel.claus@gmx.net" and I will send you the updated source code.
GeneralVS2010 Version PinmemberUR-IT11-Jun-10 6:06 
GeneralWhy not use paging to handle large datasets PinmemberDewey5-Apr-10 9:04 
GeneralRe: Why not use paging to handle large datasets [modified] Pinmemberbl@cksheep5-Apr-10 9:19 
GeneralRe: Why not use paging to handle large datasets PinmemberDewey5-Apr-10 13:53 
RantRe: Why not use paging to handle large datasets Pinmemberpvones3-Jul-10 6:56 
GeneralRe: Why not use paging to handle large datasets Pinmemberbl@cksheep3-Jul-10 11:58 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.140916.1 | Last Updated 3 Jul 2010
Article Copyright 2010 by bl@cksheep
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid