[clamav-users] Scan for dummy file with /dev/zero takes longer

[Andrew Williams]

Taizo,

The reason for the difference is that there are a lot of subsignatures used
in the published set of logical signatures that begin with some number of
zeroes (more so than the majority of random byte sequences), so the ClamAV
pattern matcher has to do a lot more work on the all-zeroes file
determining that none of the signatures fully match.  Also, there are
likely some short all-zero subsigs that get used based on how certain
subsignature features are implemented, and these can also affect
performance on large files consisting mainly of zeroes.

Hope that helps!  Thanks for asking about this - your observation is a good
reminder to us that a large all-zero file makes a good test case for
catching signatures that might have egregious performance impacts. :)

-Andrew

Andrew Williams
Malware Research Team
Cisco Talos

On Tue, Jul 9, 2019 at 11:07 PM Taizo ITO <taizo.ito at hennge.com> wrote:

> Hello,
>
> I'm trying to get some stats on how long a scan takes by different
> size, but I encountered an unexpected behavior when scanning a file
> generated in a specific way.
> A scan for a dummy file filled with /dev/zero takes much longer than
> with /dev/urandom. I think the processing time should be the same or
> less.
> I'd like to know how to avoid this problem because that may cause stuck
> service.
> I'm using ClamAV version 0.101.2/25504.
>
> You can reproduce the problem by doing the following.
>
> A 10MB file with /dev/zero: Takes almost 7 times as long as /dev/urandom.
> ```
> $ dd if=/dev/urandom of=dummy-dd-10MB-with-urandom.iso bs=10MB count=1
> $ dd if=/dev/zero of=dummy-dd-10MB-with-zero.iso bs=10MB count=1
> $ ls -ltr dummy-dd-10MB-with-*
> -rw-r--r-- 1 user user 10000000 Jul  9 03:41 dummy-dd-10MB-with-zero.iso
> -rw-r--r-- 1 user user 10000000 Jul  9 03:41 dummy-dd-10MB-with-urandom.iso
>
> $ time (echo "SCAN dummy-dd-10MB-with-zero.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-10MB-with-zero.iso: OK
> real 0m4.056s
> user 0m0.008s
> sys 0m0.004s
>
> $ time (echo "SCAN dummy-dd-10MB-with-urandom.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-10MB-with-urandom.iso: OK
> real 0m0.569s
> user 0m0.012s
> sys 0m0.000s
> ```
>
>
> A 250MB file with /dev/zero: Takes almost 8 times as long as /dev/urandom.
> ```
> $ dd if=/dev/zero of=dummy-dd-250MB-with-zero.iso bs=25MB count=10
> $ dd if=/dev/urandom of=dummy-dd-250MB-with-urandom.iso bs=25MB count=10
> $ ls -ltr dummy-dd-250MB-with-*
> -rw-r--r-- 1 user user 250000000 Jul  9 03:44
> dummy-dd-250MB-with-urandom.iso
> -rw-r--r-- 1 user user 250000000 Jul  9 03:44 dummy-dd-250MB-with-zero.iso
>
> $ time (echo "SCAN dummy-dd-250MB-with-zero.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-250MB-with-zero.iso: OK
> real 1m42.949s
> user 0m0.009s
> sys 0m0.003s
>
> $time (echo "SCAN dummy-dd-250MB-with-urandom.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-250MB-with-urandom.iso: OK
> real 0m12.905s
> user 0m0.004s
> sys 0m0.007s
> ```
>
> Thanks.
>
> --
> Taizo Ito
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20190716/d0ee60b0/attachment.htm>