Best way to process large amounts of CSV data?

Question

1.00/5 (1 vote)

See more:

I am new to coding and currently am responsible for processing large csv files in xlsx form. The data that needs to be analyzed are call detail records for investigations. I would like to develop and application for a Windows platform that would allow an average user to input these records and, post processing, display the following results: all numbers being communicated with in descending order, dates and times ranges to be manually selected based upon time frame of activity imported, ma display of all lat and long coordinates. I am currently using MS Excel for this and have not yet realized the results I was hoping for. I am not an experienced programmer and am working toward improving my skills and figured this would be a great place to start.

Thanks in advance....

What I have tried:

Excel, although I haven't figured out how to automate the process once I start analyzing the data I have. I have also tried to learn python but have not found a project simple enough for me to learn more. A .net developer told me I should try C# but I have yet to start learning that language as of yet.

Posted 24-May-17 15:22pm

jmt605

Updated 25-May-17 20:57pm

Add a Solution

Comments

Maciej Los 25-May-17 2:07am

Such a question may be answered based on opinion. That's the reason i reported it as off-topic.
In my opinion you should start learning C#.
Click here[^] to start searching CP KnowledgeBase.

Richard MacCutchan 25-May-17 3:43am

You could learn Excel macros to automate what you want to produce. Use Google to find tutorials on "Excel vba".

Maciej Los 26-May-17 2:42am

Well... As per my experience i suspect sad effects of using Excel for large amount of data. But, of course, it depends what "large amount of data" means...

Richard MacCutchan 26-May-17 2:46am

True, I have only used small to medium amounts. But, as with so many questions here, there really is no useful answer.

Maciej Los 26-May-17 3:00am

True. That's why i posted a more than generic answer ;)
Cheers,
Maciej

Robert Welliever 26-May-17 19:03pm

In my opinion, if you are a beginner your best bet is to download a free Visual Studio Express version, and start a new C# desktop app. I've handled a number of tasks similar to yours and have found using an OLEDB adapter to connect to the file will serve you best. Once read, you can manipulate it in memory as a DataTable, and you can use something like a DataGridView to allow users sorting/paging. On a side note, the 10MB file you mentioned is relatively small. Size shouldn't be a concern.

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Maciej Los · Answer 1 · 2017-05-25T20:57:00

Please, read my comments to the question.

You have to focus on C# if you want to be able to operate large amount of data comming from csv files. Of course, it depends of what "large amount of data" means in your case.

There are two - at least - ways to manipulate csv data:
1) using ADO.Net[^] with OleDb[^]
Read Text File Specific Columns[^]
OLE DB Tutorial (C#)[^]
Much ADO About Text Files[^]
Schema.ini File (Text File Driver) | Microsoft Docs[^]
How to: Add a Schema Definition to a Text File Data Source[^]
Read Text File (txt, csv, log, tab, fixed length)[^]

2) using custom parser
CSV File Parser[^]
CSV Parser (C#)[^]
Using LINQ to read delimited text files[^]

jmt605 · Answer 2 · 2017-05-26T06:56:00

Solution 2

Thanks for the input so far.  The avg file size I would need to process would be between 2.5 and 10 mb.  Being a forensic examiner I read from many sources that python is a great beginner language so I am following the pack on that.  Macros seems to be the easiest solution to learn while cutting my teeth on coding "hello world" and what is your age programs.

Posted 26-May-17 6:56am

jmt605

Updated 26-May-17 7:49am

Comments

Maciej Los 26-May-17 15:38pm

This is not an answer. Please. delete it to avoid down-voting. To post a comment, use "Have a question or comment" widget.