Click here to Skip to main content
15,887,683 members
Home / Discussions / C#
   

C#

 
GeneralRe: Implementing a LogHelper class - is this approach right Pin
Bartosz Jarmuż19-Aug-15 9:15
Bartosz Jarmuż19-Aug-15 9:15 
GeneralRe: Implementing a LogHelper class - is this approach right Pin
Eddy Vluggen19-Aug-15 11:25
professionalEddy Vluggen19-Aug-15 11:25 
AnswerRe: Implementing a LogHelper class - is this approach right Pin
V.19-Aug-15 10:43
professionalV.19-Aug-15 10:43 
GeneralWhat's more efficient in .Any and .Count in C# (Extension methods) Pin
rosharavinda19-Aug-15 5:01
rosharavinda19-Aug-15 5:01 
GeneralRe: What's more efficient in .Any and .Count in C# (Extension methods) Pin
Pete O'Hanlon19-Aug-15 5:25
mvePete O'Hanlon19-Aug-15 5:25 
GeneralRe: What's more efficient in .Any and .Count in C# (Extension methods) Pin
OriginalGriff19-Aug-15 5:32
mveOriginalGriff19-Aug-15 5:32 
GeneralRe: What's more efficient in .Any and .Count in C# (Extension methods) Pin
Nitzan Levi19-Aug-15 9:47
Nitzan Levi19-Aug-15 9:47 
QuestionPerformance of FileStream, scale is stalking me Pin
Rob Philpott19-Aug-15 3:36
Rob Philpott19-Aug-15 3:36 
A very good afternoon/time of day to all.

So, I want to read very large files sequentially. By very big, I'm thinking of 1GB up to maybe several 100GB. First question is, how sensible/practical/possible is it to have a file of say 250GB?

Probably not very sensible at all, but I'm observing strange effects even at a couple of GB. 1GB files behave the way I'd expect all the time, but 3GB files often do not.

To demonstrate what I mean, I create 8 files, the first 1GB in size, the subsequent 1GB bigger. File 1 = 1GB, file 8 = 8GB:
C#
private static void Main(string[] args)
{
    byte[] buffer = new byte[1 << 20]; //1MB
    Random r = new Random();

    for (int mb = 1024; mb <= 8192; mb += 1024)
    {
        using (FileStream fs = File.Create(string.Format(@"c:\temp\GB{0}.dat", mb >> 10)))
        {
            for (long index = 0; index < mb; index++)
            {
                r.NextBytes(buffer);
                fs.Write(buffer, 0, buffer.Length);
            }
        }
    }
}

14 minutes later I have my test files, all full of lots of randomness. And all I'm going to do is read each file in its entirety using FileStream.Read:
C#
private static void Main(string[] args)
{
    byte[] buffer = new byte[1 << 20]; //1MB
    Random r = new Random();

    for (int mb = 1024; mb <= 8192; mb += 1024)
    {
        Console.Write("GB{0}.dat: ", mb >> 10);
        Stopwatch sw = Stopwatch.StartNew();
        using (FileStream fs = File.Open(string.Format(@"c:\temp\GB{0}.dat", mb >> 10), FileMode.Open))
        {
            for (long index = 0; index < mb; index++)
            {
                fs.Read(buffer, 0, buffer.Length);
            }
        }
        Console.WriteLine("{0:0.00}s, {1:0.00}MB/s", sw.ElapsedMilliseconds / 1000d, mb * 1000d / sw.ElapsedMilliseconds);
    }
}

Now, regardless of the file size I'd expect the sequential read speed to me similar, here's what I get:
VB
GB1.dat: 17.54s, 58.38MB/s
GB2.dat: 39.20s, 52.25MB/s
GB3.dat: 149.56s, 20.54MB/s
GB4.dat: 92.97s, 44.06MB/s
GB5.dat: 175.25s, 29.22MB/s
GB6.dat: 84.29s, 72.90MB/s
GB7.dat: 104.96s, 68.29MB/s
GB8.dat: 179.43s, 45.66MB/s

First thought would be, well things slow down when other processes are accessing the disk so the inconsistency could just be other processes doing stuff, but the results seem to be a bit too consistent. At about 3GB, the speed always decreases rapidly.

I don't know what's going on. Windows is clearly caching, the disc probably is, maybe there's some interaction (garbage collection/virtual memory) which arises at certain sizes.

My second question - do you think the size of the file affects the speed which you can read it sequentially? Actually, I'd like random access but let's do the simple things first.
Regards,
Rob Philpott.

AnswerRe: Performance of FileStream, scale is stalking me Pin
Eddy Vluggen19-Aug-15 4:49
professionalEddy Vluggen19-Aug-15 4:49 
GeneralRe: Performance of FileStream, scale is stalking me Pin
Rob Philpott19-Aug-15 4:52
Rob Philpott19-Aug-15 4:52 
GeneralRe: Performance of FileStream, scale is stalking me Pin
Eddy Vluggen19-Aug-15 4:54
professionalEddy Vluggen19-Aug-15 4:54 
GeneralRe: Performance of FileStream, scale is stalking me Pin
Rob Philpott19-Aug-15 5:09
Rob Philpott19-Aug-15 5:09 
GeneralRe: Performance of FileStream, scale is stalking me Pin
Eddy Vluggen19-Aug-15 7:09
professionalEddy Vluggen19-Aug-15 7:09 
GeneralRe: Performance of FileStream, scale is stalking me Pin
Rob Philpott19-Aug-15 7:18
Rob Philpott19-Aug-15 7:18 
GeneralRe: Performance of FileStream, scale is stalking me Pin
Eddy Vluggen19-Aug-15 8:11
professionalEddy Vluggen19-Aug-15 8:11 
AnswerRe: Performance of FileStream, scale is stalking me Pin
Afzaal Ahmad Zeeshan19-Aug-15 5:41
professionalAfzaal Ahmad Zeeshan19-Aug-15 5:41 
GeneralRe: Performance of FileStream, scale is stalking me Pin
Rob Philpott19-Aug-15 5:52
Rob Philpott19-Aug-15 5:52 
GeneralRe: Performance of FileStream, scale is stalking me Pin
Afzaal Ahmad Zeeshan19-Aug-15 5:58
professionalAfzaal Ahmad Zeeshan19-Aug-15 5:58 
GeneralRe: Performance of FileStream, scale is stalking me Pin
Rob Philpott19-Aug-15 6:10
Rob Philpott19-Aug-15 6:10 
GeneralRe: Performance of FileStream, scale is stalking me Pin
Pete O'Hanlon19-Aug-15 6:18
mvePete O'Hanlon19-Aug-15 6:18 
QuestionMSBuild The type or namespace name 'Class1' could not be found (are you missing a using directive or an assembly reference?) error Pin
Txalaparta18-Aug-15 23:18
professionalTxalaparta18-Aug-15 23:18 
QuestionRe: MSBuild The type or namespace name 'Class1' could not be found (are you missing a using directive or an assembly reference?) error Pin
Eddy Vluggen19-Aug-15 0:44
professionalEddy Vluggen19-Aug-15 0:44 
AnswerRe: MSBuild The type or namespace name 'Class1' could not be found (are you missing a using directive or an assembly reference?) error Pin
Txalaparta19-Aug-15 2:05
professionalTxalaparta19-Aug-15 2:05 
GeneralRe: MSBuild The type or namespace name 'Class1' could not be found (are you missing a using directive or an assembly reference?) error Pin
Eddy Vluggen19-Aug-15 2:39
professionalEddy Vluggen19-Aug-15 2:39 
QuestionSession Ends after any change made in asp.net Pin
Salman Azhar18-Aug-15 22:10
Salman Azhar18-Aug-15 22:10 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.