Click here to Skip to main content
15,906,106 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I have a C# windows application, sometimes the application would be used to import records from excel file, i could import 2500 records in one minute using batch insertion on my development machine, when i tried normal regular insertion approach, it also took the same one minute to insert 2500 records, this get worse after i have deployed the application and installed on a client's machine, it took more than two hours to import same number of record, i just felt since i am using bulk insertion, the speed should be greatly optimised, below is my code:

What I have tried:

C#
public int InsertTaxRecordBatchTransaction(IEnumerable<EmployeeDetails> employeeDetails)
       {
           SQLiteConnection conn = null;
           SQLiteCommand cmd = null;
           var results = new List<int>();

           try
           {
               string insert = "INSERT INTO IncomeTax (OrganisationId,OrganisationName,StaffId,StaffName,Department,Position,NoOfMonthsWorked,TaxIdNumber,SessionId,Month,Year,BasicSalaryVALUES (@OrganisationId,@OrganisationName,@StaffId,@StaffName,@Department,@Position,@NoOfMonthsWorked,@TaxIdNumber,@SessionId,@Month,@Year,@BasicSalary);";
               using (conn = new SQLiteConnection(connstring))
               {
                   conn.Open();
                   using (var transaction = conn.BeginTransaction())
                   {
                       try
                       {
                           using (cmd = conn.CreateCommand())
                           {
                               //cmd = new SQLiteCommand(insert, conn);
                               cmd.CommandType = CommandType.Text;
                               cmd.CommandText = insert;
                               cmd.CommandTimeout = 900000000;
                               cmd.Parameters.Add("@OrganisationId", DbType.String, 50).Value = OrganisationId;
                               cmd.Parameters.Add("@OrganisationName", DbType.String, 50).Value = OrganisationName;
                               cmd.Parameters.Add("@StaffID", DbType.String, 50).Value = StaffId;
                               cmd.Parameters.Add("@StaffName", DbType.String, 50).Value = StaffName;
                               cmd.Parameters.Add("@Department", DbType.String, 50).Value = Department;
                               cmd.Parameters.Add("@Position", DbType.String, 50).Value = Position;
                               cmd.Parameters.Add("@NoOfMonthsWorked", DbType.Int16, 5).Value = NoOfMonthsWorked;
                               cmd.Parameters.Add("@TaxIdNumber", DbType.String, 5).Value = TaxIdNumber;
                               cmd.Parameters.Add("@SessionId", DbType.String, 20).Value = SessionId;
                               cmd.Parameters.Add("@Month", DbType.String, 20).Value = Month;
                               cmd.Parameters.Add("@Year", DbType.String, 20).Value = Year;
                               cmd.Parameters.Add("@BasicSalary", DbType.Decimal, 2).Value = BasicSalary;

                               foreach (var employee in employeeDetails)
                               {
                                   cmd.Parameters.Add("@OrganisationId", DbType.String, 50).Value = employee.OrganisationId;
                                   cmd.Parameters.Add("@OrganisationName", DbType.String, 50).Value = employee.OrganisationName;
                                   cmd.Parameters.Add("@StaffID", DbType.String, 50).Value = employee.StaffId;
                                   cmd.Parameters.Add("@StaffName", DbType.String, 50).Value = employee.StaffName;
                                   cmd.Parameters.Add("@Department", DbType.String, 50).Value = employee.Department;
                                   cmd.Parameters.Add("@Position", DbType.String, 50).Value = employee.Position;
                                   cmd.Parameters.Add("@NoOfMonthsWorked", DbType.Int16, 5).Value = employee.NoOfMonthsWorked;
                                   cmd.Parameters.Add("@TaxIdNumber", DbType.String, 5).Value = employee.TaxIdNumber;
                                   cmd.Parameters.Add("@SessionId", DbType.String, 20).Value = employee.SessionId;
                                   cmd.Parameters.Add("@Month", DbType.String, 20).Value = employee.Month;
                                   cmd.Parameters.Add("@Year", DbType.String, 20).Value = employee.Year;
                                   cmd.Parameters.Add("@BasicSalary", DbType.Decimal, 2).Value = employee.BasicSalary;

                                   results.Add(cmd.ExecuteNonQuery());
                               }

                           }
                           transaction.Commit();
                           DspMsg = "Tax computed successfully!";
                       }
                       catch(Exception ex)
                       {
                           transaction.Rollback();
                           DspMsg = ex.Message+ " some errors have occured, no record was saved! ";
                       }
                   }
               }
              // conn.Close();
           }
           catch (SQLiteException ex)
           {

               DspMsg = ex.Message;
           }
           return results.Sum();
       }
Posted
Updated 18-Apr-17 13:35pm
Comments
[no name] 18-Apr-17 19:31pm    
And you expect us to be able to tell you why one computer runs slower than another?
Uwakpeter 19-Apr-17 2:08am    
i also installed on my system to test, it also took a long time(the same two hours it took in other machines), secondary my dev machine is even lower in specs than the clients machine. so i think it is about system specification

1 solution

I'm surprised the code worked at all. You're adding NEW PARAMETER OBJECTS to the command object you're recycling on every iteration of that loop. So, at the time you execute the query, you have 12 parameters. At the end of the second iteration 24 parameters, 36 parameters, ... At the end of 2500 records, you have 30,000 parameters.

You already created the parameter objects before the loop. Too bad you're not reusing them. It would have made the code a lot faster.
 
Share this answer
 
Comments
Uwakpeter 19-Apr-17 2:18am    
I am using transaction on sqlite for the first time, i research and got this solution on code project and the solution was marked as answer (https://www.codeproject.com/Articles/853842/Csharp-Avoiding-Performance-Issues-with-Inserts-in), in any case i will appreciate if you could be a bit more clearer so i could do the necessary corrections. Thanks
Dave Kreskowiak 19-Apr-17 10:13am    
You need to go back and take a closer look at the code in that article. In your loop, you're adding new parameter objects to the commands parameters collection:
    cmd.Parameters.Add("@OrganisationId", DbType.String, 50).Value = employee.OrganisationId;

You need to reuse the existing parameter objects inside your loop:
    cmd.Parameters["@OrganisationId"].Value = employee.OrganisationId;
Uwakpeter 19-Apr-17 11:08am    
Excellent reply, it works, thanks dave. Now i can batch insert 10,000 records in 30 secs, that was great.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900