My application reads a file (normally plain text), then compresses it, then creates a bitmap from the compressed data.
Up to a file size of about 30 MB of text (approximately 15 MB compressed) it works just as expected.
At 40 MB of text (20 MB compressed) I sometimes get an exception when creating the new bitmap. Anything above that errors every time in the same way.
Source is 30 MB Text File, compressed to approximately 15 MB
Image Height 2943
Image Width 5232
pixels = 15397776
No Error
Source is 40MB Text file, compressed to approximately 20 MB
ImageHeight 3402 Integer
ImageWidth 6048 Integer
pixels = 20575296
Offending Line - MainBitmap = New Bitmap(ImageWidth, ImageHeight)
System.ArgumentException was unhandled
HResult=-2147024809
Message=Parameter is not valid.
Source=System.Drawing
The user starts by either opening a text file or creating / typing a password. Once both are done, the user selects "Encrypt" and the process begins:
1 - The text file contents (stored as an array of bytes) is compressed to another byte array using either GZIP or Deflate compression.
2 - The compressed array size is translated to a Bitmap Width and Height with approximate Aspect Ratios as the user chooses (16:9, 4:3, 1:1), and an empty bitmap is created
3 - For each possible byte in the array (0..255), multiple unique colors are generated and added to a list (of color), using the SHA Hash of the character and the password concatenated in a fancy manner.
4 - for each byte in the text file contents, the bitmap pixel is set to one of the colors that corresponds to that byte.
5 - When finished, the user can save the bitmap as a .png file (lossless)
Step two is where the exception occurs
The Bitmap is declared at Form level just as a Bitmap, then set as a new bitmap once the user chooses to encrypt a bunch of text. At that point The Memory Profile details show the 50 MB array as 52 MB and the compressed as 25. Memory usage spikes very high (900 MB) when the text is loaded, then settles back down to 80 MB or so and stays there. It rises a small amount just before the exception.
The "original" array is the size of the text file, using File.ReadAllBytes.
The compressed array is around half the size of the "original" array.
I have two Dictionaries that are 1024 entries each (Integer,Color) and (Color,Integer)
What I have tried:
<pre lang="text">
I converted the Forms App to a Console App and it breezes through a 100 to 200 MB source file with no issues at all.
I also rewrote the encryption part in the Forms App and got rid of every "large object" (except the bitmap)
The input file never gets read into an array or string. The input file is compressed file to file into a compressed file. If the Input File is less than 256K, it gets displayed, otherwise only the first and last lines are displayed using a streamreader to read lines, again, no big strings.
That compressed file is directly saved to disk as a Temp File, it never gets into an array.
When encrypting, one byte at a time is binary read from the compressed file and processed.
Tested and confirmed working (by decrypting) on anything up to and including 40 MiB.
At 50 MiB, the same error occurs every time while setting the bitmap size (Invalid Parameter).
It's not a gradual error, meaning I can try a 50 MB file, and get an error, then try 1,2,5,10,20 etc MB files with no errors, then 50 again, which errors.
Here is the exact code except I have replaced 2 dictionaries with one List, it handles 50 MB now but not 60. On my x64 machine it handles 200 but shows 900 MB of memory used.
Option Strict On
Imports System.IO
Imports System.IO.Compression
Imports System.Security.Cryptography
Imports System.Text
Public Class Form1
Private WithEvents TxtStatus As New TextBox
Private WithEvents BtnTest As New Button
Private WithEvents TxtEncryptKey As New TextBox
Private Label1 As New Label
Private WithEvents BtnLoadFile As New Button
Private TxtPlain As New RichTextBox
Private RNG As New Random
Private CompressedFileName As String
Private CompressedLength As Long
Private InputFileName As String
Private InputLength As Long
Private ColorList As New List(Of Color)
Private MainBitmap As Bitmap
Private EncryptKey As String
Private ImageWidth As Integer
Private ImageHeight As Integer
Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
AutoScaleMode = AutoScaleMode.Font
ClientSize = New Size(604, 270)
MinimumSize = Size
With TxtStatus
.Anchor = CType(((AnchorStyles.Bottom Or AnchorStyles.Left) Or AnchorStyles.Right), AnchorStyles)
.Location = New Point(4, 224)
.Multiline = True
.Size = New Size(594, 37)
End With
With BtnTest
.Location = New Point(206, 188)
.Size = New Size(120, 23)
.Text = "Test"
End With
With TxtEncryptKey
.Anchor = CType(((AnchorStyles.Top Or AnchorStyles.Left) Or AnchorStyles.Right), AnchorStyles)
.Location = New Point(80, 131)
.Size = New Size(518, 22)
End With
With Label1
.AutoSize = True
.Location = New Point(3, 134)
.Size = New Size(70, 14)
.Text = "Password:"
End With
With BtnLoadFile
.Location = New Point(80, 188)
.Size = New Size(120, 23)
.Text = "Load A File"
End With
With TxtPlain
.Anchor = CType(((AnchorStyles.Top Or AnchorStyles.Left) Or AnchorStyles.Right), AnchorStyles)
.Location = New Point(4, 2)
.Size = New Size(594, 123)
End With
With Me
.Controls.Add(TxtPlain)
.Controls.Add(BtnLoadFile)
.Controls.Add(BtnTest)
.Controls.Add(TxtStatus)
.Controls.Add(Label1)
.Controls.Add(TxtEncryptKey)
.Font = New Font("Consolas", 9.0!, FontStyle.Regular, GraphicsUnit.Point, CType(0, Byte))
.Text = "ONLY FOR TESTING " & " " & Application.ProductVersion & " TESTING ONLY"
End With
End Sub
Private Function GetSHA1Color(ByVal Input As String) As Color
Dim SHA1Hasher As New SHA1Managed()
Dim Hash As Byte() = SHA1Hasher.ComputeHash(Encoding.Default.GetBytes(Input))
Return Color.FromArgb(255, Hash(0), Hash(1), Hash(2))
End Function
Private Function FillColorList(EncryptKey As String) As Boolean
ColorList.Clear()
Dim ColorToAdd As Color = Nothing
For CharacterValue As Integer = 0 To 255
ColorToAdd = GetSHA1Color(EncryptKey & Chr(CharacterValue))
If ColorList.Contains(ColorToAdd) Then
Return False
Else
ColorList.Add(ColorToAdd)
End If
Next
Return True
End Function
Private Sub CompressFileToFile(InFile As String, Outfile As String)
Dim InFileInfo As FileInfo = New FileInfo(InFile)
Dim InFileLength As Long = InFileInfo.Length
Dim Bytesread As Long = 0
Dim BytesRemaining As Long = InFileLength
Dim BufferSize As Long = InFileLength \ 100
Dim BytesToRead As Long = BufferSize
Dim buffer(CInt(BufferSize - 1)) As Byte
Dim OutFileinfo As New FileInfo(Outfile)
Me.Cursor = Cursors.WaitCursor
Try
Using originalFileStream As FileStream = InFileInfo.OpenRead()
Using compressedFileStream As FileStream = File.Create(Outfile)
Using compressionStream As New GZipStream(compressedFileStream, CompressionMode.Compress)
Do While BytesRemaining > 0
originalFileStream.Read(buffer, 0, CInt(BufferSize))
compressionStream.Write(buffer, 0, CInt(BufferSize))
Bytesread += BufferSize
TxtStatus.Text = "Compressing - " & ((Bytesread * 100) \ InFileLength).ToString("000") & "%"
TxtStatus.Refresh()
BytesRemaining -= BufferSize
If BytesRemaining < BufferSize Then BufferSize = BytesRemaining
Loop
End Using
End Using
End Using
Catch ex As Exception
TxtStatus.Text = "Error Compressing Input File " & ex.Message
If File.Exists(CompressedFileName) Then File.Delete(CompressedFileName)
End Try
Me.Cursor = Cursors.Default
End Sub
Private Sub CalcSize(InputString As String)
CompressedFileName = Path.GetTempFileName
CompressFileToFile(InputString, CompressedFileName)
If Not File.Exists(CompressedFileName) Then
InputFileName = ""
CompressedFileName = ""
MainBitmap = Nothing
Exit Sub
End If
Dim FileLength As Integer = CInt(My.Computer.FileSystem.GetFileInfo(CompressedFileName).Length)
CompressedLength = FileLength
Dim Temp As Long = 0
For N As Integer = 1 To 100000
ImageWidth = N
ImageHeight = N
Temp = ImageWidth * ImageHeight
If Temp > FileLength Then
Exit For
End If
Next
TxtStatus.Text = "Compressed Length is " & FileLength.ToString & ", the bitmap will be " & ImageWidth.ToString & " by " & ImageHeight.ToString & vbNewLine
If Not MainBitmap Is Nothing Then
MainBitmap.Dispose()
End If
Try
MainBitmap = New Bitmap(ImageWidth, ImageHeight, Imaging.PixelFormat.Format32bppArgb)
Catch ex As Exception
TxtStatus.Text = ImageWidth.ToString & "x" & ImageHeight.ToString & " Bitmap Creation Failed - " & ex.Message & vbNewLine & ex.StackTrace
MainBitmap = Nothing
End Try
End Sub
Private Sub BtnTest_Click(sender As System.Object, e As System.EventArgs) Handles BtnTest.Click
If EncryptKey = "" Then
TxtStatus.Text = "A Password is required."
Exit Sub
End If
If Not File.Exists(InputFileName) OrElse My.Computer.FileSystem.GetFileInfo(InputFileName).Length = 0 Then
TxtStatus.Text = "Some Plain text is required"
Exit Sub
End If
If Not FillColorList(EncryptKey) Then
MessageBox.Show("The Password you chose : " & EncryptKey & vbNewLine &
"Cannot be used for testing." & vbNewLine &
"Choose a different password.", "Cannot test with this password", MessageBoxButtons.OK, MessageBoxIcon.Error)
Exit Sub
End If
TxtStatus.Text = "Preparing .........."
TxtStatus.Refresh()
CalcSize(InputFileName)
If File.Exists(CompressedFileName) Then File.Delete(CompressedFileName)
If MainBitmap Is Nothing Then
Exit Sub
Else
TxtStatus.Text = ImageWidth.ToString & "x" & ImageHeight.ToString & " Bitmap Creation Succeeded"
End If
End Sub
Private Sub TxtKey_TextChanged(sender As System.Object, e As System.EventArgs) Handles TxtEncryptKey.TextChanged
EncryptKey = TxtEncryptKey.Text
BtnTest.Enabled = EncryptKey.Length > 0
End Sub
Private Sub BtnLoadFile_Click(sender As System.Object, e As System.EventArgs) Handles BtnLoadFile.Click
Dim FileSize As Long = 0
Dim TempString As String = ""
Using OFD As New OpenFileDialog
OFD.Title = "Find the text file you wish to encrypt"
OFD.Filter = "Text Files (*.txt)|*.txt"
If OFD.ShowDialog = Windows.Forms.DialogResult.OK Then
InputFileName = OFD.FileName
FileSize = My.Computer.FileSystem.GetFileInfo(InputFileName).Length
InputLength = FileSize
Try
If FileSize > 64 * 1024 Then
Dim firstLine As String = Nothing ' or String.Empty
Dim lastLine As String = Nothing ' same here
Using reader As New StreamReader(InputFileName)
If Not reader.EndOfStream Then firstLine = reader.ReadLine
Do Until reader.EndOfStream
lastLine = reader.ReadLine
Loop
End Using
TxtPlain.Text = firstLine &
vbNewLine & vbNewLine & "Text Redacted" & vbNewLine & vbNewLine &
lastLine
Else
TxtPlain.Text = File.ReadAllText(InputFileName, Encoding.Default)
End If
TxtStatus.Text = "Read " & FileSize.ToString & " Characters"
Catch ex As Exception
TxtStatus.Text = "Error reading file - " & ex.Message
InputFileName = ""
End Try
End If
End Using
End Sub
End Class
Submit an article or tip