Click here to Skip to main content
15,887,676 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
Hi,

I have a csv file that contain anything from a few hundred records to 50,000 records, although I have the process working the time taking to complete is painfully slow, so was looking for advice on best way of performing the requirement.

The scenario is an order can contain anything upto 50,000 codes, the order number is not present in the upload data so cannot perform a bulk insert (or i don't know how).

The CSV file only has one field such as

abcd123
efgh456


Any advise or links would be appreciated.

What I have tried:

The ID for the order is passed in the controller together with the file details. Controller method below.

[HttpPost]
public ActionResult UploadNewMailingData(int id,HttpPostedFileBase attachmentcsv)
{
    CsvFileDescription csvFileDescription = new CsvFileDescription
    {
        SeparatorChar = ',',
        FirstLineHasColumnNames = true
    };
    CsvContext csvContext = new CsvContext();
    StreamReader streamReader = new StreamReader(attachmentcsv.InputStream);
    IEnumerable<MediaSaleRecord> list = csvContext.Read<MediaSaleRecord>(streamReader, csvFileDescription);
    MediaSaleRecord mediaSaleRecord =  new  MediaSaleRecord();
    List<MediaSaleRecord> myList = new List<MediaSaleRecord>();
    for (int n = 0;n<list.Count();n++)
    {
        mediaSaleRecord.CustomerID = list.ElementAt(n).CustomerID;
        mediaSaleRecord.MediaSaleID = id;
        myList.Add(mediaSaleRecord);
    }
    db.MediaSaleRecords.AddRange(myList);
    db.SaveChanges();
    return Redirect("Index");
}


credit for the above code goes to
Upload CSV (other Delimiter-Separated) Files to Sql Server in Asp.Net MVC - Using EF Code First[^]
The model is defined is simple and only consist of:

    public class MediaSaleRecord
    {
        public int MediaSaleRecordID { get; set; }
        public int MediaSaleID { get; set; }
        public string CustomerID { get; set; }
    }
}
Posted
Updated 15-Jul-17 23:07pm

1 solution

An alternative might be CsvHelper: CsvHelper[^] but I doubt that this will be faster.

The BCP utility can be tuned using a formatting file, see: 'More Columns in Table than in Data File' in: Working with the bcp Command-line Utility - Simple Talk[^]
But I'm afraid this does not allow to define constant values.

The best solution I think is to use a 'staging table' and then insert into destination table using a INSERT INTO DEST_TABLE SELECT ... SQL query, see: SQL Server Forums - Specifying constant value for column in bcp.[^]
 
Share this answer
 

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900