1. The problem may be in
.ReadAllText
method which is not optimized by definition.
You may try to use
StreamReader.ReadLine
method. Take a look at this
discussion.
The proposed there code runs fast even on a very large text files (i tested).
2. It may be useful to check an open source code of
ClamWin antivirus and read the article about it:
Hash-AV: Fast Virus Signature Scanning from Stanford.edu.