Creating large data in C# data table perfomance very slow.

Question

0.00/5 (No votes)

See more:

Dear Friend,

In my Asp.Net c# application

we are creating datatable with 10K records and 120 columns, it is tacking more than 2 hours to complete the datatable.

THe following is the code i am using please help me out

C#

foreach (DataColumn dc in policyLevelTable.Columns)
{
    for (int i = 1; i <= 10000; i++)
    {
        if (dc.ColumnName.ToLower() == "policyid")
        {
            DataRow newdr = policyLevelTable.NewRow();
            newdr[0] = i;
            // newdr[1] = 1;
            policyLevelTable.Rows.Add(newdr);
        }
        else
        {
            DataRow[] dr = policyLevelTable.Select("policyid in (" + i + ")");
            dr[0][dc.ColumnName] = RandomParameterType(new List<int>{1,2,3,4}, 0, false);
        }
    }
}

Thanks,
Ramu.

What I have tried:

foreach (DataColumn dc in policyLevelTable.Columns)
{
for (int i = 1; i <= 10000; i++)
{
if (dc.ColumnName.ToLower() == "policyid")
{
DataRow newdr = policyLevelTable.NewRow();
newdr[0] = i;
// newdr[1] = 1;
policyLevelTable.Rows.Add(newdr);

}
else
{
DataRow[] dr = policyLevelTable.Select("policyid in (" + i + ")");

dr[0][dc.ColumnName] = RandomParameterType(new List<int> { 1, 2, 3, 4 }, 0, false);

}
}
}

Posted 4-Nov-16 1:23am

Ramug10

Updated 4-Nov-16 1:43am

#realJSOP

v2

Add a Solution

Comments

Philippe Mori 5-Nov-16 11:21am

You wrote twice the same code. Once in a properly formatted code block and once as unformatted text. What is the purpose of doing that?

3 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

OriginalGriff · Answer 1 · 2016-11-04T01:40:00

Do it the other way round: loop through the rows as the outer loop, then fill in each column inside it. That way, you don't need to do the (probably time consuming) .Select for each cell in each row, you can omit it entirely, since you will always be working on the same row.
In addition, unless you need a new List each time, create one outside all loops.
Then check RenadomParamaterType and see what it's doing, and how much of the time it's taking.

F-ES Sitecore · Answer 2 · 2016-11-04T01:43:00

Solution 3

Creating 10k rows is never going to be quick. What do you expect the user to do with 10k rows anyway? If they spend 3 seconds per row it would take them 8 hours to process all of the rows. Showing that much data is useless, implement paging and searching so that you only show a reduced amount of data, or only the data that is relevant.

Posted 4-Nov-16 1:43am

F-ES Sitecore

v2

Comments

#realJSOP 4-Nov-16 11:25am

The "why" of it is not relevant, IMHO.

#realJSOP · Answer 3 · 2016-11-04T01:43:00

I don't know how long this is gonna take, but there's less code, so silver linings...

C#

Random random = new Random();
for (int i = 1; i &lt;= 10000; i++)
{
    DataRow newdr = policyLevelTable.NewRow();
    for (int j = 0; j < policyLevelTable.Columns.Count; i++)
    {
        newdr[j] = (j==0) ? j : random.Next(5);
    }
}

Caveat - this is going to consume quite a bit of memory. If this is done on a website, you will likely have more problems than simple speed issues.

Creating large data in C# data table perfomance very slow.

3 solutions

Solution 1

Solution 3

Solution 2

Add your solution here

Preview 0