[clamav-users] Scanning very large files in chunks

Al Varnell alvarnell at mac.com
Thu Aug 4 22:14:05 EDT 2016


Does anybody have any evidence of malware that exceeds 4GB? Although I can certainly see the utility of the proposed capability as a hedge for the future, it would seem to be a waste of time and compute power to scan such large files today.

With the ever increasing malware issues we face today, it’s important to consider this:

Risk = threat x vulnerability x consequence

<http://fortune.com/2016/05/14/cybersecurity-risk-calculation/>

We all need to focus on fixing the high risk items first.

Sent from Janet's iPad

-Al-

On Aug 4, 2016, at 4:40 PM, sapientdust+clamav at gmail.com wrote:
> 
> I've recently run into the issue of clamd not being able to scan files
> that are larger than a small number of GB, and I have seen the
> warnings in `man clamd.conf` that say specified limits above 4GB are
> ignored.
> 
> Could developers or other folks familiar with the clamd codebase
> comment on the feasibility of scanning large files in multiple pieces
> as a way of handling larger files?
> 
> For example, given a file that is 6GB, does using multiple INSTREAM
> calls (that's how I'm interacting wth clamd currently) to check the
> full 6GB seem like it should work reliably?
> 
> INSTREAM: bytes 0-1000MB
> INSTREAM: bytes 900MB-1.9GB
> INSTREAM: bytes 1.8GB-2.8GB
> INSTREAM: bytes 2.7GB-3.7GB
> INSTREAM: bytes 3.6GB-4.6GB
> INSTREAM: bytes 4.5GB-5.5GB
> INSTREAM: bytes 5.4GB-6.0GB
> 
> There is overlap above, wherein the 100MB of data that starts at the
> 900MB position is scanned twice, once in the first call (as the last
> 100MB of that stream) and once in the second call (as the first 100MB
> of that stream), to reduce the possibility of a virus being split into
> two pieces and therefore not recognized.
> 
> If Clamav needs the first bytes in order to know
> what kind of file it is scanning and trigger filetype-specific heuristics, then
> something like the above could be adapted so that the first N bytes of the
> first chunk are prepended to each subsequent chunk that is checked for
> that file.
> 
> Thanks for any guidance or feedback you can provide.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2370 bytes
Desc: not available
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20160804/363f7d82/attachment.bin>


More information about the clamav-users mailing list