Click here to Skip to main content
13,199,680 members (68,389 online)
Click here to Skip to main content
Add your own
alternative version

Stats

19.1K views
26 bookmarked
Posted 19 Sep 2009

SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

, 19 Sep 2009
Rate this:
Please Sign up or sign in to vote.
Ever wondered how hard it would be to load a CSV file into a SQLite database?

Ever wondered how hard it would be to load a CSV file into a SQLite database. I know how I would do it in code, no rocket science needed there! However in this case I wanted to really know the speed of doing this natively and really didn’t want to code anything!

So looking at what SQLite3.exe has too offer it pretty much supports it out of the box. Very nice :-)

Requirements:

  • Loading speed
  • Making the data to consuming applications available asap

While I love C# and frankly its hard to go back to C or C++, sometimes performance trumps the creature comforts we have become accustomed to.

Note: I did this without circling back to a C# implementation as I know the data and performance requirements  are tight and in this case I wanted max performance with no code! The biggest factor to a successful implementation is to ensure you use the tools best for the job, not just the ones you favor in that specific year.

So first things first – create a table to take the input

DROP TABLE IF EXISTS BookSales;
CREATE TABLE IF NOT EXISTS BookSales
(
   Store    int
  ,Date     varchar
  ,OrderReference varchar
  ,Line     int
  ,BookISBN varchar(14)
  ,Quantity int
  ,Price    int
, Primary Key (OrderReference,Line)
);

Next is the magic. We need to load the CSV into the table:

.separator "|"
.import BookSales.txt BookSales

Wow that was easy :-) . You can see we set the separator to be a pipe rather than comma in this case, then the import.

.IMPORT [FileName] [Table]

Now the database is ready to be queried! But if we want to take it just one stage further:

.output SummaryBookSales.csv
SELECT Store, Date, BookISBN, SUM(Quantity), SUM(Price)
FROM BookSales
GROUP BY Store, Date, BookISBN;

Now we output the results of our simple aggregation into a pipe separated output file.

Tying this all together in a single configuration file, which we will call “BookAnalysisLoader.sql”, gives us:

DROP TABLE IF EXISTS BookSales;
CREATE TABLE IF NOT EXISTS BookSales
(
   Store    int
  ,Date     varchar
  ,OrderReference varchar
  ,Line     int
  ,BookISBN varchar(14)
  ,Quantity int
  ,Price    int
, Primary Key (OrderReference,Line)
);

.separator "|"
.import BookSales.txt BookSales

.output SummaryBookSales.csv
SELECT Store, Date, BookISBN, SUM(Quantity), SUM(Price)
FROM BookSales
GROUP BY Store, Date, BookISBN;
.exit

The last piece of the puzzle is the final execution:

sqlite3.exe BookSalesAnalysis.db3 < BookAnalysisLoader.sql

Now we have a newly created database with our analysis data in it, and we have a summary CSV file generated from the output. So we can load the CSV into Excel or another DB, or directly interrogate the DB for more analytical information – and all without coding!

Related Links:

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

GarethI
United States United States
I'm Gareth and am a guy who loves software! My day job is working for a retail company and am involved in a large scale C# project that process large amounts of data into up stream data repositories.

My work rule of thumb is that everyone spends much more time working than not, so you better enjoy what you do!

Needless to say - I'm having a blast.

Have fun,

Gareth

You may also be interested in...

Pro

Comments and Discussions

 
QuestionPart 1 - 7 ? Pin
Tachyonx22-Sep-16 2:44
memberTachyonx22-Sep-16 2:44 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.171020.1 | Last Updated 20 Sep 2009
Article Copyright 2009 by GarethI
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid