Signature hash by byte?

Question

0.00/5 (No votes)

See more:

Currently my Signature database in text format is over one million lines which is the reason as why my scan speeds have decreased. I am sure that I need to implement some different more efficient code to achieve better scan speeds as well but my question is how might I scan using md5 hash or crc32 hash by byte so that it will only read the first few bytes before it knows its a match or not to then quickly move to the next file? also how might I do binary search? or scan with this method?

Posted 30-Aug-11 17:35pm

Dale 2012

Updated 30-Aug-11 17:37pm

v2

Add a Solution

Comments

Sergey Alexandrovich Kryukov 31-Aug-11 0:34am

Scan of what..?
--SA

Dale 2012 31-Aug-11 1:09am

virus signatures in a text file. I would like to know how to use them more efficiently so that my scan speeds will increase. What method would you approach and how might I learn to apply it? byte or binary lookup?

Dale 2012 31-Aug-11 1:11am

My virus scanner is slowing down due to the size of my virus definition list being over one million and probably the method of my recursion is a bit weak so I am looking for some examples to achieve faster scanning times by applying some kind of algorithm to my recursive search. Please help

Dale 2012 31-Aug-11 2:50am

I seen a small example such as

(Hash ex) b11b177e7244624410406a8b26430648
(byte ex) 0xb1 0x1b 0x17 0x7e 0x72 0x44 0x62 0x44 0x10 and so forth

can I write code that will match the bytes and if the first byte or up to the second byte do not match then go on to the next file?

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

OriginalGriff · Answer 1 · 2011-08-30T21:18:00

Solution 1

MD5 and SHA hashes do not work like that - they are a complete summary across all the bytes in the file.
What you are trying to do is like adding the first three integers of a 1000 integer sequence and expecting to be able to tell something about the resulting total from that - it just doesn't work.

Posted 30-Aug-11 21:18pm

OriginalGriff

Comments

Dale 2012 31-Aug-11 4:08am

ok if not by byte then can i reference the lines of hash by binary? if not will you give me some idea of what methods to use to make scanning of large file files fast and small files even faster or the best way to use hash virus definitions without slowing down scan times?

OriginalGriff 31-Aug-11 4:14am

No - a hash does not work on bit lines, the way an XOR does - it looks at the whole data stream.
Speed up? Off the top of my head, I don't know if you can make significant inroads without changing to C++ or possibly assembler.

Dale 2012 31-Aug-11 7:36am

Thank you for your response maybe I can leave you with a post that was given on my last question concerning my problem if you will care to clarify what he may be talking about in laymen terms

So, you're reading your entire hash file tables on every file you're "scanning"? No wonder it's so slow.

You load the hash tables ONCE and keep all that data in an internal table, proabably sorted by hash code to make lookups faster.
Permalink
Posted 24 Jul '11
Dave Kreskowiak

I believe that this is the right track so what I have done is created a few text boxes that load all the contents from my md5.txt and crc32.txt which is what i thought he meant by internal table. I am not sure how it is sorted by hash code or how to sort it by hash to make look ups faster?

OriginalGriff 31-Aug-11 8:46am

What Dave is talking about is loading the hash tables from your MD5 and CRC32 (Why are you using two hashes? Particularly when CRC32 is, well, crap nowadays) once, and once only and retaining the has tables in memory for the life of your app.
If you are indeed reading them each time you want to check a file, then that could contribute quite a bit to any delays.
In addition, if you are using File.ReadAllBytes to read each individual file and check it against the hash table, it might be worth your considering changing to a native routine which handles that without reallocating buffer space unless the existing space is too small - re-use could help speed things up, because it removes the need for the Garbage Collector to shuffle things around.

Dale 2012 2-Sep-11 2:12am

What is re-use? I would like to be able to implement this into my scan procedure as what your saying makes sense. thanks in advance

Dave Kreskowiak 31-Aug-11 12:11pm

I vaguely remember this one. If I remember correctly, your "database" of hashes is a hash of every "good" file on the machine, not of virus infections. Doing it this way would result in a couple of millions of file hashes in your database. Am I correct?

Dale 2012 2-Sep-11 2:10am

sorry dave that was incorrect my database is of known viruses only

Yogesh Saharan 14-Sep-11 5:18am

That was a nice ans by Dave..I was also committing the same mistake.I was using a timer to start this scanning process and by default the interval was set to 100ms..I reduced it to 1ms and the scan time reduced a lot.Check that if it can be of any help to u.And buddy plz give me ur mail-Id..I am also building an AV and maybe u can help me a lot.Mine is [DELETED]@gmail.com...Thx

Yogesh Saharan 14-Sep-11 5:40am

It's saharan(dot)1628(atthetrate)gmail(dot)com...Need help bro