Reading large file line by line - memory mgmt and performance

Question

0.00/5 (No votes)

See more:

I'm trying to figure out the best way to read a large text file (>5GB) line by line in Python. Each line will be processed sequential too (e.g. slicing the string and pushing it to some function).

Am wondering can this be done using parallel threads / multithread in Python to make it run faster?? Also I would like to minimize the memory footprint since there are other processes running.

Any help or push in the right direction is much appreciated.

What I have tried:

python readline, readlines and streaming

Posted 3-Jan-20 6:30am

tony joseph david

Updated 3-Jan-20 7:04am

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Dave Kreskowiak · Answer 1 · 2020-01-03T07:05:00

Since a file can only be read by one thread, you'd have to have a component doing all the reading and doling out the lines to process to your processing threads.

Keep in mind, that multi-threading does not automatically make some process go faster. It depends on the problem and the amount of processing you're doing, how long each pass in the processing takes, what the bottlenecks are in the process, ...

If the major bottleneck is I/O (reading the file from whatever device or external source), threading the processing isn't going to do anything for you. Your threads are going to be stalled waiting for something to process.