How to fetch thousand records in 100*10 batches

Question

1.00/5 (1 vote)

See more:

Hi Friends,

I need to pull the data from the SQL Db and do some calculations. I am not interested to fetch all the records in a one shot. My requirement is to run the loop in batches.
For example if suppose i have 1050 records. it should run 11 loops ie 10*100+ 50

Whats the method should i follow. Should i write a stored procedure. If yes please give me some idea.

Below i my snapshot of the code

C#

SqlDataAdapter daAuthors = new SqlDataAdapter("Select * From Authors", objConn);
DataSet dsPubs = new DataSet("Pubs");
daAuthors.Fill(dsPubs,"Authors");

DataTable tblAuthors;
tblAuthors = dsPubs.Tables["Authors"];

			foreach (DataRow drCurrent in tblAuthors.Rows)
			{
				Console.WriteLine("{0} {1}",
					drCurrent["au_fname"].ToString(),
					drCurrent["au_lname"].ToString());
			}

Posted 11-May-15 9:39am

jinesh sam

Add a Solution

Comments

PIEBALDconsult 11-May-15 16:54pm

What database system?
If SQL Server 2012 (or newer), you might want to use the offset_fetch clause:
https://msdn.microsoft.com/en-us/library/ms188385(v=sql.110).aspx

And don't use a DataAdapter.

Maciej Los 11-May-15 16:54pm

What kind of calculation? Have you tried Common Table Expressions?

jinesh sam 11-May-15 21:36pm

@PIEBALDconsult I am Using SQL Server 2008. I accept your point i will use datareader

jinesh sam 11-May-15 21:41pm

@Maciej Los Calculations is little complex.Thats the reason, i am planning to do in batches.All calculations are on front end not on server side. I think CTE is for server side

1 solution

Add a Solution

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sascha Lefèvre · Answer 1 · 2015-05-11T10:16:00

I'd suggest a third option: Query for all records at once, but don't use DataAdapter.Fill - instead use a DataReader to read row by row. That way you don't have to deal with select-paging while still not having to read all records at once. It will even have the best performance (only one query and using a DataReader is faster than DataAdapter.Fill).

SqlDataReader (MSDN)[^]
http://www.dotnetperls.com/sqldatareader[^]

edit: It will have the best performance if the calculation has, for some reason, to be done in the application. If it could be done within the query (CTE) that would most certainly be the fastest way. But we can't tell, because we don't know what your calculation involves (yet).

edit2 after comment: In that case (processing millions of records and having to compare the results to a DB2 database) batching seems indeed to be the best approach. As you're using SQL Server 2008, there's no OFFSET FETCH-clause yet, so you will have to use some some 'conventional' paging approach. I think this article explains your options well:
SQL Server 2005 Paging Results[^]
Otherwise there are even more to take a look at:
https://www.google.com/search?q=codeproject+sql+server+select+paging&ie=utf-8&oe=utf-8[^]

Nonetheless, you could use a DataReader instead of DataAdapter.Fill to improve the performance somewhat.

How to fetch thousand records in 100*10 batches

1 solution

Solution 1

Add your solution here

Preview 0

How to fetch thousand records in 100*10 batches

1 solution

Solution 1

Add your solution here

Preview 0

Existing Members

...or Join us