Click here to Skip to main content
       

Database

 
AnswerRe: Database design question PinmemberPIEBALDconsult27-Dec-12 16:52 
AnswerRe: Database design question Pinmemberjschell28-Dec-12 11:16 
GeneralRe: Database design question PinmemberPIEBALDconsult28-Dec-12 15:42 
GeneralRe: Database design question Pinmemberjschell30-Dec-12 9:28 
AnswerRe: Database design question PinmemberMycroft Holmes28-Dec-12 12:05 
QuestionImporting of Huge fixed width text file in sql server table Pinmemberibrahim sharief27-Dec-12 10:24 
AnswerRe: Importing of Huge fixed width text file in sql server table PinmemberPIEBALDconsult27-Dec-12 15:08 
AnswerRe: Importing of Huge fixed width text file in sql server table PinmemberDeepak Kr130-Dec-12 21:29 
AnswerRe: Importing of Huge fixed width text file in sql server table Pinmemberjschell31-Dec-12 10:14 
Questionshow Sum(Balance) as ob on Rows Header and add next rows balance in ob Pinmemberhadeed147227-Dec-12 7:52 
AnswerRe: show Sum(Balance) as ob on Rows Header and add next rows balance in ob Pinmemberdjj5527-Dec-12 8:53 
GeneralRe: show Sum(Balance) as ob on Rows Header and add next rows balance in ob Pinmemberhadeed147227-Dec-12 9:03 
QuestionSQL BulkCopy using IDataReader - how can you cast the fields? PinmemberAllan Watkins26-Dec-12 20:12 
AnswerRe: SQL BulkCopy using IDataReader - how can you cast the fields? PinmemberMycroft Holmes26-Dec-12 21:32 
Well I would say you are screwed, only because you are trying to transform the data in your load. IMHO transforms are the biggest time waster ever inflicted on the load process. I would split the operations to a load process and then a transform. Caveat I am not addressing the size issue as I have never had the problem (and 500mb is very small beer in our environment).
 
I would do the following.
Read in the data file and get the header record.
Create a staging table in sql server exactly matching the column headers - every column to be varchar or nvarchar if needed
Convert the csv file to a datatable (This article [^]may help).
Use BulkCopy to load the data (everything is varchar so it WILL load)
Use a stored proc to do the transforms from the staging table to the target table. If you have to use a dictionary file then you are going to have to take that into account and it will be a challenge (probably a crap load of dynamic sql).
Drop the staging table
Never underestimate the power of human stupidity
RAH

GeneralRe: SQL BulkCopy using IDataReader - how can you cast the fields? PinmemberAllanW1111127-Dec-12 7:02 
AnswerRe: SQL BulkCopy using IDataReader - how can you cast the fields? PinmemberPIEBALDconsult27-Dec-12 9:48 
QuestionSubquery returns more than 1 row PinmemberJassim Rahma24-Dec-12 2:02 
AnswerRe: Subquery returns more than 1 row PinmemberEddy Vluggen24-Dec-12 2:29 
AnswerRe: Subquery returns more than 1 row PinmemberMaulikDusara24-Dec-12 22:44 
GeneralRe: Subquery returns more than 1 row PinmemberEddy Vluggen25-Dec-12 3:41 
GeneralRe: Subquery returns more than 1 row PinmemberMycroft Holmes26-Dec-12 12:17 
GeneralRe: Subquery returns more than 1 row PinmemberMaulikDusara26-Dec-12 19:45 
GeneralRe: Subquery returns more than 1 row PinmemberEddy Vluggen27-Dec-12 5:24 
QuestionOn the Naming of Columns for Lookup tables. PinmemberBrady Kelly17-Dec-12 23:44 
AnswerRe: On the Naming of Columns for Lookup tables. PinmemberMycroft Holmes18-Dec-12 0:19 
GeneralRe: On the Naming of Columns for Lookup tables. PinmemberBrady Kelly18-Dec-12 0:26 
GeneralRe: On the Naming of Columns for Lookup tables. PinmemberMycroft Holmes18-Dec-12 2:11 
GeneralRe: On the Naming of Columns for Lookup tables. PinmemberBrady Kelly18-Dec-12 2:15 
AnswerRe: On the Naming of Columns for Lookup tables. PinmemberPIEBALDconsult18-Dec-12 5:05 
GeneralRe: On the Naming of Columns for Lookup tables. PinmemberBrady Kelly18-Dec-12 5:12 
GeneralRe: On the Naming of Columns for Lookup tables. PinmemberPIEBALDconsult18-Dec-12 5:46 
GeneralRe: On the Naming of Columns for Lookup tables. PinmemberBrady Kelly18-Dec-12 6:34 
QuestionOn Strategies to Spell Check the Structure of a Database. PinmemberBrady Kelly14-Dec-12 4:02 
JokeRe: On Strategies to Spell Check the Structure of a Database. PinmemberChris Meech14-Dec-12 5:31 
AnswerRe: On Strategies to Spell Check the Structure of a Database. PinmemberSimon_Whale14-Dec-12 5:52 
AnswerRe: On Strategies to Spell Check the Structure of a Database. PinmemberMycroft Holmes14-Dec-12 14:12 
GeneralRe: On Strategies to Spell Check the Structure of a Database. PinmemberBrady Kelly14-Dec-12 17:36 
GeneralRe: On Strategies to Spell Check the Structure of a Database. PinmemberMycroft Holmes14-Dec-12 17:57 
GeneralRe: On Strategies to Spell Check the Structure of a Database. PinmemberBrady Kelly14-Dec-12 18:12 
GeneralRe: On Strategies to Spell Check the Structure of a Database. PinmemberMycroft Holmes14-Dec-12 22:13 
GeneralRe: On Strategies to Spell Check the Structure of a Database. PinmemberBrady Kelly14-Dec-12 22:25 
GeneralRe: On Strategies to Spell Check the Structure of a Database. PinmemberPIEBALDconsult15-Dec-12 9:49 
GeneralRe: On Strategies to Spell Check the Structure of a Database. PinmemberBrady Kelly15-Dec-12 19:12 
GeneralRe: On Strategies to Spell Check the Structure of a Database. PinmemberPIEBALDconsult18-Dec-12 11:47 
GeneralRe: On Strategies to Spell Check the Structure of a Database. PinmemberBrady Kelly18-Dec-12 12:01 
Questionsending images from a SQL database to a user's email as attachment PinmemberMember 946798213-Dec-12 3:16 
QuestionRe: sending images from a SQL database to a user's email as attachment PinmemberEddy Vluggen13-Dec-12 3:29 
QuestionODBC Returns Wrong Data PinmemberGenJerDan12-Dec-12 6:23 
AnswerRe: ODBC Returns Wrong Data PinmemberChris Meech12-Dec-12 6:50 
GeneralRe: ODBC Returns Wrong Data PinmemberGenJerDan12-Dec-12 7:20 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.


Advertise | Privacy | Mobile
Web04 | 2.8.141220.1 | Last Updated 21 Dec 2014
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Service
Layout: fixed | fluid