Best way to read 1GB file (Line by Line) and save data temporary on any location without cache memory usage

Question

1.00/5 (2 votes)

See more:

I am working on desktop application using C# win from. I need to read line by line 1GB file and parse it and after that I need to display all parse data on chart.

What are the best steps for it?

Main problem is, I need to save data temporary because, in future, I will use parse data again.

Posted 21-Sep-15 3:22am

Farah Uzma

Updated 21-Sep-15 3:50am

Naveen.Sanagasetti

v2

Add a Solution

Comments

Maciej Los 21-Sep-15 12:31pm

Parse and parse again... Please provide more details about data structure. Is there any chance to read data via using OleDb provider?

Farah Uzma 21-Sep-15 13:46pm

I am working with winform application using C#.I want to read a txt file( size not fix it can be 2GB) and after parsing lines , save parse data on temporary location because ,I will be used that data for other task. what will be best solution to save parse data ?

I have tested two solutions:

1)datatable but it takes cache memory which is hanging application for further task.
2)Access db,but it is slow process, because every time we call ExecuteNonQuery() after parsing line.

Maciej Los 21-Sep-15 13:55pm

You did not provide information abot data structure...

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Patrice T · Answer 1 · 2015-09-21T09:19:00

Quote:
without cache memory usage

You have no choice in this matter, memory cache usage is mandatory. It is that way because cache gives huge benefits even to read one time a huge file.
So you can put your file and temporary file about anywhere on a HDD.

Last time a user was having choice about cache, is back to MS-DOS.

Without knowing what is in the first file, what parse and what usage of the filtered file you have after, it is impossible to give advice on an efficient technique for your problem.