Multi - Section Downloads TEMP files

Question

0.00/5 (No votes)

See more:

, +

Hi guys!

I'm creating an application for multi-segment downloading (many threads per single download) ...but I have a BIG QUESTION:

My program analyzes the remote file and creates segments/sections with equal ranges
so each thread downloads its own segment...Say, I want to download a file 70MB in size and
my program creates two threads to download ranges ( 0 - 35 MB; 35 - 70 MB)..
Each thread saves downloaded content in its OWN TEMPORARY FILE...

When download is complete my program simply merges those files, creates the final resulting FILE of them and when done DELETES those temporary files...

Is it effective to have several files per download?? IS it more effective to have one
temporary file and to use LOCKS to allow only one thread write downloaded content
in time and than try to sort bytes somehow and create the resulting file???

How do you think what choice is better???

I personally think that when I use many files per download I can easily merge them
cause my program tracks each segment and assigns index to it...so when download
is complete it sorts those segments by index and merges files easily BUT on other
side many files and many threads make hard drive to switch between hard drive clusters and tracks...

PLEASE HELP...

Posted 10-Oct-11 14:06pm

Nick Reshetinsky

Updated 10-Oct-11 16:37pm

v7

Add a Solution

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Ganesan Senthilvel · Answer 1 · 2011-10-10T17:22:00

Nick,

My personal choice would be running each thread per file. No need to have multiple threads for single file. Becaz of the application requirement.

Your application needs to download multiple files at the same time. Right? In that scenario, thread per file is the better option. Key reason is performance. CPU clock cycle is added up due to the file split algorithm + multi thread management for the single file + later merging. I don't think it will greatly add the performance value due to multithread per file.

Ray Radin · Answer 2 · 2011-10-10T23:19:00

Hello Nick,

I learned a lot creating my own downloader some time ago and I'd like to offer some suggestions.

First, 1 thread per download.
Create any number of connections you need for the segments in the same thread.
I actually tried 1 thread per segment, totally unnecessary.

Second, 1 file per download would be easier and requires no locking if you also use 1 thread per download.
Managing many files per download can be challenging, and if the download is big, merging would take some time.

Good luck with your code.

Multi - Section Downloads TEMP files

2 solutions

Solution 1

Solution 2

Add your solution here

Preview 0