[clamav-users] Input Stream Scanning for very large files

Dennis Peterson dennispe at inetnw.com
Fri Jan 25 21:43:19 EST 2019


You can easily use the unix split command and cat to scan files of any size. Or 
use perl to break stream file segments to the stream. The first file in a split 
or segment contains the file time and will need to be concatenated to the 
beginning of each split or segment so clamav knows what it is. It doesn't matter 
if the file makes no sense just so long as no malware is found. You will need 
two split sizes in order to ensure a signature doesn't span splits which means 
at least two runs of each large file, but that is trivial when scripted. SSD 
drives would be useful.

dp

On 1/25/19 10:12 AM, Paul Kosinski wrote:
> I understand that it's impractical for ClamAV to scan exceedingly large
> files, as it could fill up RAM and/or page forever. But the current 4GB
> hard limit is overly restrictive, especially since 32-bit addresses and
> numbers are ancient history in current OSes.
>
> In particular, scanning big archives immediately after downloading is
> desirable, since there can be malware that attacks the de-compressing or
> de-archiving mechanism itself. Thus simply scanning the individual
> contained files isn't completely adequate.
>
> Is there any plan to allow scanning bigger files? There still are,
> after all, size limits specifiable in the config file(s), and warnings
> about the consequences of setting limits too big can be documented.
>
>
> On Fri, 25 Jan 2019 14:32:55 +0000
> "Micah Snyder (micasnyd)" <micasnyd at cisco.com> wrote:
>
>> Re
garding specific limits, I'm sorry to say that ClamAV is presently
>> limited to max file size of 4GB on most systems (and, I think
>> unintentionally, 2GB on some systems).
>>
>> -Micah
>>
>>
>> On Jan 24, 2019, at 4:23 PM, J.R.
>> <themadbeaker at gmail.com<mailto:themadbeaker at gmail.com>> wrote:
>>
>> I think I framed my problem statement differently.
>> So, our requirement is similar the one asked by John in the
>> below link. I do not know if the solution proposed is a correct one..
>> Also, how do you propose I should scan an archive of 100GB ( let's
>> say) size. Does clamav have any limitations on scanning a single file
>> of such huge size ??
>>
>> Without knowing more about this "archive" it's hard to say if ClamAV
>> will even pick up anything, due to the reason Micah gave in his reply.
>> But another issue is if this is just one humongous file you are trying
>> to shove through and say it *does* trigger some virus... How are you
>> going to know what / where the virus is? All you know is its somewhere
>> in your massive archive file...
>>
>> You would be much better off scanning the individual files as you
>> assemble said archive, and obviously only need to scan files where an
>> infection would make sense (i.e. a text file isn't going to contain a
>> virus)...
>>
>> There are stream settings in the clamd.conf, but I don't know what the
>> hard upper-limits are.
>>
>> In cases like this, it's probably best to assemble you own sample
>> archives, one clean & one infected, and run through your proposed
>> process. If it works as intended, then create a few more samples and
>> re-test... If it doesn't work as intended then you'll need to re-think
>> your process...
> _______________________________________________
> clamav-users mailing list
> clamav-users at lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml





More information about the clamav-users mailing list