Click here to Skip to main content
13,703,316 members
Rate this:
 
Please Sign up or sign in to vote.
See more:
My application reads a file (normally plain text), then compresses it, then creates a bitmap from the compressed data.
Up to a file size of about 30 MB of text (approximately 15 MB compressed) it works just as expected.
At 40 MB of text (20 MB compressed) I sometimes get an exception when creating the new bitmap. Anything above that errors every time in the same way.

Source is 30 MB Text File, compressed to approximately 15 MB
Image Height 2943
Image Width 5232
pixels = 15397776
No Error

Source is 40MB Text file, compressed to approximately 20 MB
ImageHeight 3402 Integer
ImageWidth 6048 Integer
pixels = 20575296
Offending Line - MainBitmap = New Bitmap(ImageWidth, ImageHeight)
System.ArgumentException was unhandled
HResult=-2147024809
Message=Parameter is not valid.
Source=System.Drawing

The user starts by either opening a text file or creating / typing a password. Once both are done, the user selects "Encrypt" and the process begins:
1 - The text file contents (stored as an array of bytes) is compressed to another byte array using either GZIP or Deflate compression.
2 - The compressed array size is translated to a Bitmap Width and Height with approximate Aspect Ratios as the user chooses (16:9, 4:3, 1:1), and an empty bitmap is created
3 - For each possible byte in the array (0..255), multiple unique colors are generated and added to a list (of color), using the SHA Hash of the character and the password concatenated in a fancy manner.
4 - for each byte in the text file contents, the bitmap pixel is set to one of the colors that corresponds to that byte.
5 - When finished, the user can save the bitmap as a .png file (lossless)
Step two is where the exception occurs

The Bitmap is declared at Form level just as a Bitmap, then set as a new bitmap once the user chooses to encrypt a bunch of text. At that point The Memory Profile details show the 50 MB array as 52 MB and the compressed as 25. Memory usage spikes very high (900 MB) when the text is loaded, then settles back down to 80 MB or so and stays there. It rises a small amount just before the exception.
The "original" array is the size of the text file, using File.ReadAllBytes.
The compressed array is around half the size of the "original" array.
I have two Dictionaries that are 1024 entries each (Integer,Color) and (Color,Integer)

What I have tried:

<pre lang="text">
I converted the Forms App to a Console App and it breezes through a 100 to 200 MB source file with no issues at all.

I also rewrote the encryption part in the Forms App and got rid of every "large object" (except the bitmap)
The input file never gets read into an array or string. The input file is compressed file to file into a compressed file. If the Input File is less than 256K, it gets displayed, otherwise only the first and last lines are displayed using a streamreader to read lines, again, no big strings.
That compressed file is directly saved to disk as a Temp File, it never gets into an array.
When encrypting, one byte at a time is binary read from the compressed file and processed.
Tested and confirmed working (by decrypting) on anything up to and including 40 MiB.
At 50 MiB, the same error occurs every time while setting the bitmap size (Invalid Parameter).
It's not a gradual error, meaning I can try a 50 MB file, and get an error, then try 1,2,5,10,20 etc MB files with no errors, then 50 again, which errors.

Here is the exact code except I have replaced 2 dictionaries with one List, it handles 50 MB now but not 60. On my x64 machine it handles 200 but shows 900 MB of memory used.

Option Strict On
Imports System.IO
Imports System.IO.Compression
Imports System.Security.Cryptography
Imports System.Text

Public Class Form1
    Private WithEvents TxtStatus As New TextBox
    Private WithEvents BtnTest As New Button
    Private WithEvents TxtEncryptKey As New TextBox
    Private Label1 As New Label
    Private WithEvents BtnLoadFile As New Button
    Private TxtPlain As New RichTextBox
    Private RNG As New Random
    Private CompressedFileName As String
    Private CompressedLength As Long
    Private InputFileName As String
    Private InputLength As Long
    Private ColorList As New List(Of Color)
    Private MainBitmap As Bitmap
    Private EncryptKey As String
    Private ImageWidth As Integer
    Private ImageHeight As Integer

    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
        AutoScaleMode = AutoScaleMode.Font
        ClientSize = New Size(604, 270)
        MinimumSize = Size
        With TxtStatus
            .Anchor = CType(((AnchorStyles.Bottom Or AnchorStyles.Left) Or AnchorStyles.Right), AnchorStyles)
            .Location = New Point(4, 224)
            .Multiline = True
            .Size = New Size(594, 37)
        End With
        With BtnTest
            .Location = New Point(206, 188)
            .Size = New Size(120, 23)
            .Text = "Test"
        End With
        With TxtEncryptKey
            .Anchor = CType(((AnchorStyles.Top Or AnchorStyles.Left) Or AnchorStyles.Right), AnchorStyles)
            .Location = New Point(80, 131)
            .Size = New Size(518, 22)
        End With
        With Label1
            .AutoSize = True
            .Location = New Point(3, 134)
            .Size = New Size(70, 14)
            .Text = "Password:"
        End With
        With BtnLoadFile
            .Location = New Point(80, 188)
            .Size = New Size(120, 23)
            .Text = "Load A File"
        End With
        With TxtPlain
            .Anchor = CType(((AnchorStyles.Top Or AnchorStyles.Left) Or AnchorStyles.Right), AnchorStyles)
            .Location = New Point(4, 2)
            .Size = New Size(594, 123)
        End With

        With Me
            .Controls.Add(TxtPlain)
            .Controls.Add(BtnLoadFile)
            .Controls.Add(BtnTest)
            .Controls.Add(TxtStatus)
            .Controls.Add(Label1)
            .Controls.Add(TxtEncryptKey)
            .Font = New Font("Consolas", 9.0!, FontStyle.Regular, GraphicsUnit.Point, CType(0, Byte))
            .Text = "ONLY FOR TESTING " & " " & Application.ProductVersion & " TESTING ONLY"
        End With

    End Sub

    Private Function GetSHA1Color(ByVal Input As String) As Color
        Dim SHA1Hasher As New SHA1Managed()
        Dim Hash As Byte() = SHA1Hasher.ComputeHash(Encoding.Default.GetBytes(Input))
        Return Color.FromArgb(255, Hash(0), Hash(1), Hash(2))
    End Function

    Private Function FillColorList(EncryptKey As String) As Boolean
        ColorList.Clear()
        Dim ColorToAdd As Color = Nothing
        For CharacterValue As Integer = 0 To 255
            ColorToAdd = GetSHA1Color(EncryptKey & Chr(CharacterValue))
            If ColorList.Contains(ColorToAdd) Then
                Return False
            Else
                ColorList.Add(ColorToAdd)
            End If
        Next
        Return True
    End Function

    Private Sub CompressFileToFile(InFile As String, Outfile As String)
        Dim InFileInfo As FileInfo = New FileInfo(InFile)
        Dim InFileLength As Long = InFileInfo.Length
        Dim Bytesread As Long = 0
        Dim BytesRemaining As Long = InFileLength
        Dim BufferSize As Long = InFileLength \ 100
        Dim BytesToRead As Long = BufferSize
        Dim buffer(CInt(BufferSize - 1)) As Byte
        Dim OutFileinfo As New FileInfo(Outfile)
        Me.Cursor = Cursors.WaitCursor
        Try
            Using originalFileStream As FileStream = InFileInfo.OpenRead()
                Using compressedFileStream As FileStream = File.Create(Outfile)
                    Using compressionStream As New GZipStream(compressedFileStream, CompressionMode.Compress)
                        Do While BytesRemaining > 0
                            originalFileStream.Read(buffer, 0, CInt(BufferSize))
                            compressionStream.Write(buffer, 0, CInt(BufferSize))
                            Bytesread += BufferSize
                            TxtStatus.Text = "Compressing - " & ((Bytesread * 100) \ InFileLength).ToString("000") & "%"
                            TxtStatus.Refresh()
                            BytesRemaining -= BufferSize
                            If BytesRemaining < BufferSize Then BufferSize = BytesRemaining
                        Loop
                    End Using
                End Using
            End Using
        Catch ex As Exception
            TxtStatus.Text = "Error Compressing Input File " & ex.Message
            If File.Exists(CompressedFileName) Then File.Delete(CompressedFileName)
        End Try
        Me.Cursor = Cursors.Default
    End Sub

    Private Sub CalcSize(InputString As String)
        CompressedFileName = Path.GetTempFileName
        CompressFileToFile(InputString, CompressedFileName)
        If Not File.Exists(CompressedFileName) Then
            InputFileName = ""
            CompressedFileName = ""
            MainBitmap = Nothing
            Exit Sub
        End If
        Dim FileLength As Integer = CInt(My.Computer.FileSystem.GetFileInfo(CompressedFileName).Length)
        CompressedLength = FileLength
        Dim Temp As Long = 0
        For N As Integer = 1 To 100000
            ImageWidth = N
            ImageHeight = N
            Temp = ImageWidth * ImageHeight
            If Temp > FileLength Then
                Exit For
            End If
        Next
        TxtStatus.Text = "Compressed Length is " & FileLength.ToString & ", the bitmap will be " & ImageWidth.ToString & " by " & ImageHeight.ToString & vbNewLine
        If Not MainBitmap Is Nothing Then
            MainBitmap.Dispose()
        End If
        Try
            MainBitmap = New Bitmap(ImageWidth, ImageHeight, Imaging.PixelFormat.Format32bppArgb)
        Catch ex As Exception
            TxtStatus.Text = ImageWidth.ToString & "x" & ImageHeight.ToString & " Bitmap Creation Failed - " & ex.Message & vbNewLine & ex.StackTrace
            MainBitmap = Nothing
        End Try
    End Sub

    Private Sub BtnTest_Click(sender As System.Object, e As System.EventArgs) Handles BtnTest.Click
        If EncryptKey = "" Then
            TxtStatus.Text = "A Password is required."
            Exit Sub
        End If

        If Not File.Exists(InputFileName) OrElse My.Computer.FileSystem.GetFileInfo(InputFileName).Length = 0 Then
            TxtStatus.Text = "Some Plain text is required"
            Exit Sub
        End If

        If Not FillColorList(EncryptKey) Then
            MessageBox.Show("The Password you chose : " & EncryptKey & vbNewLine &
                            "Cannot be used for testing." & vbNewLine &
                            "Choose a different password.", "Cannot test with this password", MessageBoxButtons.OK, MessageBoxIcon.Error)
            Exit Sub
        End If

        TxtStatus.Text = "Preparing .........."
        TxtStatus.Refresh()
        CalcSize(InputFileName)
        If File.Exists(CompressedFileName) Then File.Delete(CompressedFileName)
        If MainBitmap Is Nothing Then
            Exit Sub
        Else
            TxtStatus.Text = ImageWidth.ToString & "x" & ImageHeight.ToString & " Bitmap Creation Succeeded"
        End If

    End Sub

    Private Sub TxtKey_TextChanged(sender As System.Object, e As System.EventArgs) Handles TxtEncryptKey.TextChanged
        EncryptKey = TxtEncryptKey.Text
        BtnTest.Enabled = EncryptKey.Length > 0
    End Sub

    Private Sub BtnLoadFile_Click(sender As System.Object, e As System.EventArgs) Handles BtnLoadFile.Click
        Dim FileSize As Long = 0
        Dim TempString As String = ""
        Using OFD As New OpenFileDialog
            OFD.Title = "Find the text file you wish to encrypt"
            OFD.Filter = "Text Files (*.txt)|*.txt"
            If OFD.ShowDialog = Windows.Forms.DialogResult.OK Then
                InputFileName = OFD.FileName
                FileSize = My.Computer.FileSystem.GetFileInfo(InputFileName).Length
                InputLength = FileSize
                Try
                    If FileSize > 64 * 1024 Then
                        Dim firstLine As String = Nothing ' or String.Empty
                        Dim lastLine As String = Nothing ' same here

                        Using reader As New StreamReader(InputFileName)
                            If Not reader.EndOfStream Then firstLine = reader.ReadLine
                            Do Until reader.EndOfStream
                                lastLine = reader.ReadLine
                            Loop
                        End Using
                        TxtPlain.Text = firstLine &
                                        vbNewLine & vbNewLine & "Text Redacted" & vbNewLine & vbNewLine &
                                        lastLine
                    Else
                        TxtPlain.Text = File.ReadAllText(InputFileName, Encoding.Default)
                    End If
                    TxtStatus.Text = "Read " & FileSize.ToString & " Characters"
                Catch ex As Exception
                    TxtStatus.Text = "Error reading file - " & ex.Message
                    InputFileName = ""
                End Try
            End If
        End Using
    End Sub
End Class
Posted 10-Nov-17 13:28pm
Updated 17-Nov-17 6:24am
v2

1 solution

Rate this: bad
 
good
Please Sign up or sign in to vote.

Solution 1

Simple, you can't make a Bitmap object that big, especially so if you're forgetting to call Dispose on the object that require it when you're done using them. This is especially true of drawing objects.

Read this[^],
  Permalink  
Comments
zomalaja 10-Nov-17 20:18pm
   
As I stated, the console version has no issues at all. The bitmap is never displayed anywhere, it exists solely to be saved as a png file. I'm not sure what you mean by calling dispose on the object that require it. What types of object would that be other than a picturebox or something where the bitmap gets displayed/painted ?
The link you provided is talking about bitmaps that are 21000x21000, several orders bigger than what I'm trying.
Dave Kreskowiak 10-Nov-17 21:05pm
   
Bitmaps also need to be disposed. Just about ALL the GDI objects need to be disposed. If you're making Bitmap after Bitmap after Bitmap and not disposing them when you're done with them, you're app is leaking resources and memory.

The point of that link wasn't the size, it was how Bitmap objects work in GDI and the limitations imposed by memory.
Dave Kreskowiak 10-Nov-17 22:25pm
   
That's cute and all, and completely useless. Your test code is meaningless as it's not doing what your app is doing. We either see the real code or you're on your own.

What you missed in the link is that memory fragmentation can also affect the limits of the size of the images you can create with GDI.
Dave Kreskowiak 11-Nov-17 8:44am
   
If the variable that is supposed to be holding the Bitmap is null, there's nothing to Dispose. It wasn't created.

But, if you're reusing the same variable over and over to hold new Bitmaps, the old Bitmap MUST be Disposed before you create the new one to replace it.

Something you're doing is fragmenting memory to the point where you can't create a Bitmap big enough. A Bitmap is not the only large object you're creating. If you're reading the entire file into memory before you're doing anything with it, you're creating a large object. If you're hold the content of the encrypted file in memory, you're creating another large object. If you're using a List<> and keep adding and adding to it, you're creating large object after large object because the internal array in the List must be enlarged and the content copied to the new array.

Also, what's with creating the Bitmap in the first place? What could you possibly be showing in it that has anything to do with compressing a text file?
Dave Kreskowiak 11-Nov-17 18:51pm
   
There is no facility for attachments or images. I'm also not downloading untrusted code from an untrusted source. You paste the relevant bits of code into your original post.
zomalaja 27-Nov-17 20:26pm
   
Thanks for nothing - hopefully someone will see my request and remove this thread.
zomalaja 24-Nov-17 12:08pm
   
Attn any Moderator - if possible please delete this entire thread, it has been a total waste of my time and none of the information I was given is the least bit useful.

Thanks in advance.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
Top Experts
Last 24hrsThis month


Advertise | Privacy | Cookies | Terms of Service
Web01-2016 | 2.8.180920.1 | Last Updated 17 Nov 2017
Copyright © CodeProject, 1999-2018
All Rights Reserved.
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100