Load Huge data in a editable control

This is not a product or full source to use, but a nice tutorial of how to do so. C++ is used in these tutorials (not C#).


http://www.catch22.net/tuts/memory-techniques-part-1[^]
http://www.catch22.net/tuts/memory-techniques-part-2[^]

Use in Index in SQL.
You Get Fast Data Access.
For Eg.
Defining Indexes with SQL Server 2005[^]

As other have mentionned, you have to use virtual mode when loading lot of data and you have to ensure that not all data is cached.

I don't know how DatagridView virtual mode works (since I uses Infragistics grids) but you might have to cache some rows depending on the inner working of the control and how much time it takes to get a row.

Sometime I uses a cache of 65536 rows that contains last row for a given 65536 modulo of the index. It is easy to compute and keep a relatively large number of rows in memory so control does not have to get data too often.

However given that a screen has typically less than 1000 available pixel in height and you might display around 50 rows at a time, it become impossible to view all items by only dragging the scrollbar around 50000 items. Arrow can still be used to handle larger set of data but for it become very painfull to find specific items when the grid approach 1 million rows.

Not only that but for very large data set, you might have to disable some features like sorting as they might become very slow... or you might have to implement that outside of the UI (for exemple at the database level).

Anyway for such large data (250 million rows), you cannot rely only on existing controls. If a row would require only 10 bytes, you would need 2.5 GB of RAM to load all in memory.

You have to uses some common sense and profiling might also help to get an idea of preformance of memory usage.

In my opinion for such large data, you have to split the data somehow (paging) or do some filtering (show most relevant items). If the data is based on time, then you should probably split the data at regular interval like displaying a single day. Otherwise, you might consider doing paging and load say 10000 rows at a time. At that level, you should not have performance problem and it is still very usable for a user (easy to view all data by scrolling with the thumb box)