[ c# ] outofmemory io exception @ streamwriter/binarywriter/strings

Question

1.00/5 (1 vote)

See more:

Hello,
I wanted to know why i get a OutOfMemory Exception in the following:
basicly I have a textbox/string with ~ 90.000.000 characters (or more)
And what I want do is writing them to a file..

C#

BinaryWriter writer = new BinaryWriter(new FileStream(AppDomain.CurrentDomain.BaseDirectory + "test.txt", FileMode.Create, FileAccess.Write));
writer.Write(textBox1.Text);
writer.Close();

C#

using (StreamWriter outfile = new StreamWriter(AppDomain.CurrentDomain.BaseDirectory + "test.txt"))
{
    outfile.Write(textBox1.Text);
}

So on this way I get the memory exceptions.

If I try to reduce the size by splitting the main string into
subs ..

C#

if (textBox1.TextLength >= 28000000)
{
    int step_size = 28000000;
    int step_count = textBox1.TextLength / step_size;

    int cur_begin = 0;
    int cur_end = step_size;

    textBox4.AppendText("TextBoxCharacterCount: " + textBox1.TextLength.ToString() + Environment.NewLine);
    for (int i = 0; i < step_count; i++)
    {

        string sub = textBox1.Text.Substring(cur_begin, cur_end);
        if (i == 0)
        {

            BinaryWriter writer = new BinaryWriter(new FileStream(AppDomain.CurrentDomain.BaseDirectory + textBox2.Text + ".h", FileMode.Create, FileAccess.Write));
            writer.Write(sub);
            writer.Close();
        }
        else
        {
            BinaryWriter writer = new BinaryWriter(new FileStream(AppDomain.CurrentDomain.BaseDirectory + textBox2.Text + ".h", FileMode.Open, FileAccess.Write));
            writer.Write(sub);
            writer.Close();
        }

        cur_begin = cur_end + 1;
        cur_end = cur_end + step_size - 1;
    }

    string end = textBox1.Text.Substring(cur_begin, textBox1.TextLength);
    BinaryWriter end_writer = new BinaryWriter(new FileStream(AppDomain.CurrentDomain.BaseDirectory + textBox2.Text + ".h", FileMode.Open, FileAccess.Write));
    end_writer.Write(end);
    end_writer.Close();
}
else
{
    BinaryWriter writer = new BinaryWriter(new FileStream(AppDomain.CurrentDomain.BaseDirectory + textBox2.Text + ".h", FileMode.Create, FileAccess.Write));
    writer.Write(textBox1.Text);
    writer.Close();
}

I still get the memory exception at sub this time.
But why ? a string can contain 2.147.483.647 characters in c#
and the current size would be far below, means ~ 28.000.000

And the most important fact is, that if i want write a string with
~ 29.000.000 with the very first method, it works , but if i try to split
90.000.000 into 28.000.000 parts it doesnt work at all ...

,greetings

What I have tried:

destribes above in the related question :)

Posted 15-Feb-16 11:48am

GAFO

Add a Solution

Comments

Sergey Alexandrovich Kryukov 15-Feb-16 17:48pm

So what? Eventually, you face some data size which is not suitable for available memory. Isn't it a 32-bit application?
You may think of not reading the whole file at once, using a database, and so on...
—SA

GAFO 15-Feb-16 18:01pm

well, i need to read a whole file since i need to convert a .xm file into a byte array -
works fine until the size of the array goes above the 29.000.000 limit - and i am not as much into c# , since im a c++ coder

Sascha Lefèvre 15-Feb-16 18:34pm

Any file already is a byte array. Either you have a misconception here or you didn't express yourself clearly.

GAFO 15-Feb-16 18:39pm

BinaryReader reader = new BinaryReader(new FileStream(openFileDialog1.FileName, FileMode.Open, FileAccess.Read));
byte[] ImageData = reader.ReadBytes((int)reader.BaseStream.Length);
reader.Close();

string hex = BitConverter.ToString(ImageData).Replace("-", ", 0x");
int count = hex.Count(f => f == 'x');
count = count + 1;

To make myself clear, this is reading all the bytes of an file into an byte array, then it get moved into an string and - replaced with the hex style of e.g. 0xA3 , this string is way too long for writing as it seems, even if c# should manage the memory on its own compared to c++ , Hope its not clear, the lenght of this string would be around 90.000.000 characters within the string. (or much more) At the end this string has to be written into a file, thats basicly all which should be done

3 solutions

Solution 1

First, why are you using BinaryWriter to write a text file?

It's a lot less code to just do this and end up treating the files as text files:

File.WriteAllText("someFilePath.h", TextBox1.Text);

Posted 15-Feb-16 12:49pm

Dave Kreskowiak

Comments

GAFO 15-Feb-16 17:59pm

I can tell you why -> http://i.imgur.com/2KCmo6W.png its too big as it seems

Dave Kreskowiak 15-Feb-16 18:01pm

Using BinaryWriter is not going to solve this at all.

On which line does the exception throw?

GAFO 15-Feb-16 18:03pm

the exception is directly at the line of File.WriteAllText :)
isnt there a way to write large (realy huge) strings into a file without oom ?

Sascha Lefèvre 15-Feb-16 18:22pm

Did you compile your project for x86? If yes, give "Any CPU" a try (which will result in x64 if you have a 64bit CPU).

GAFO 15-Feb-16 18:26pm

it is at "Any CPU" and I thought that
c# is supposed to manage its memory automaticly, thatswhy Im using it in this case , normaly there shouldnt be any outofmemory error O.o

GAFO 15-Feb-16 18:34pm

I might have an idea to solve this problem but not enought knowledge in c# >.<
The string looks like this (pattern .. much longer in original)
"static unsigned char binarray[] = { 0x45, 0x78, 0x74, 0x65, 0x6E, 0x64, 0x65, 0x64, 0x20, 0x4D, 0x6F, 0x64, 0x75, 0x6C, 0x65, 0x3A, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20 ..... };"

So is there a way to split it all e.g. 1.000.000 spaces once ?
Because string.split(" "); would split it at every string and the iterating would take
ages .. :o Gues if I could split the string, all 1.000.000 spaces once, I would have
on a bytecount of 90.000.000 e.g a string array of 90 lines (just as an examble) and then i could write each entry of the array step by step

Solution 3

You should loop with much smaller block than that. Something between 8KB and 64 KB might be adequate.

Also, you should not stream and writer in each iteration. This is very basic code optimization. This is not premature optimization but just a good habit to write efficient code.

Also, you should use using statement when creating an object that is disposable. Your code is not exception safe!

Exception safety in C# – more than just trying and catching[^]

Then you have off-by one errors. In some case, your code won't write last partial block.

Also also code duplication. Have you never heard of: Don't repeat yourself - Wikipedia, the free encyclopedia[^]

So you should not repeat code that write data (3 places) and also not repeat code that compute target file name. This code is poorly reusable. Cut and paste programming is a very bad habit.

Finally, you are not given proper argument to string.Substring(Int32, Int32). If would be a good idea to read document or at least Intellisense information while writing code.

String.Substring Method (Int32, Int32) (System)[^]

Solution 2 code would be a good starting point for the structure although I'm not sure that he write data as you expect it.

Posted 15-Feb-16 16:00pm

Philippe Mori

Comments

GAFO 16-Feb-16 14:06pm

thank you for providing this suggestion, and Solution 2 was working very nice :)
and about the reuseability of the code - its a quick tool for a simple purpose, means
the structuring etc is not important in this case. Ofc I could have made classes etc to manage every single part, but that would be too much for this case ;) anyways the information in case of error handling was good to know for bigger projects in the future in case of c# and .Net

Add a Solution

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sascha Lefèvre · Accepted Answer · 2016-02-15T15:05:00

After clarifying the ultimate purpose of this as per the comments above:

There's no need to read the whole file at once. You can read and convert it in chunks.

This should do what you want; I tested it with a 100MB file:

C#

string inputFile = @"...";
string outputFile = inputFile + ".txt";
int bufferSize = 50000;

using (var reader = new BinaryReader(new FileStream(inputFile, FileMode.Open, FileAccess.Read)))
{
    if (reader.BaseStream.Length > 0)
    {
        using (var writer = new StreamWriter(outputFile, append: false, encoding: Encoding.ASCII))
        {
            writer.Write("0x");
            writer.Write(reader.ReadByte().ToString("X2"));

            byte[] buffer = new byte[bufferSize];

            while (true)
            {
                int remainingBytes = (int)(reader.BaseStream.Length - reader.BaseStream.Position);
                int bytesToRead = Math.Min(bufferSize, remainingBytes);

                if (bytesToRead == 0)
                    break;

                if (bytesToRead == bufferSize)
                    reader.Read(buffer, 0, bytesToRead);
                else
                    buffer = reader.ReadBytes(bytesToRead);

                string hex = BitConverter.ToString(buffer).Replace("-", ", 0x");
                writer.Write(", 0x");
                writer.Write(hex);
            }
        }
    }
}

[ c# ] outofmemory io exception @ streamwriter/binarywriter/strings

3 solutions

Solution 1

Solution 2

Solution 3

Add your solution here

Preview 0

[ c# ] outofmemory io exception @ streamwriter/binarywriter/strings

3 solutions

Solution 1

Solution 2

Solution 3

Add your solution here

Preview 0

Existing Members

...or Join us