Taizo,

The reason for the difference is that there are a lot of subsignatures used in the published set of logical signatures that begin with some number of zeroes (more so than the majority of random byte sequences), so the ClamAV pattern matcher has to do a lot more work on the all-zeroes file determining that none of the signatures fully match.  Also, there are likely some short all-zero subsigs that get used based on how certain subsignature features are implemented, and these can also affect performance on large files consisting mainly of zeroes.

Hope that helps!  Thanks for asking about this - your observation is a good reminder to us that a large all-zero file makes a good test case for catching signatures that might have egregious performance impacts. :)

-Andrew

Andrew Williams
Malware Research Team
Cisco Talos

On Tue, Jul 9, 2019 at 11:07 PM Taizo ITO <taizo.ito@hennge.com> wrote:
Hello,

I'm trying to get some stats on how long a scan takes by different
size, but I encountered an unexpected behavior when scanning a file
generated in a specific way.
A scan for a dummy file filled with /dev/zero takes much longer than
with /dev/urandom. I think the processing time should be the same or
less.
I'd like to know how to avoid this problem because that may cause stuck service.
I'm using ClamAV version 0.101.2/25504.

You can reproduce the problem by doing the following.

A 10MB file with /dev/zero: Takes almost 7 times as long as /dev/urandom.
```
$ dd if=/dev/urandom of=dummy-dd-10MB-with-urandom.iso bs=10MB count=1
$ dd if=/dev/zero of=dummy-dd-10MB-with-zero.iso bs=10MB count=1
$ ls -ltr dummy-dd-10MB-with-*
-rw-r--r-- 1 user user 10000000 Jul  9 03:41 dummy-dd-10MB-with-zero.iso
-rw-r--r-- 1 user user 10000000 Jul  9 03:41 dummy-dd-10MB-with-urandom.iso

$ time (echo "SCAN dummy-dd-10MB-with-zero.iso" | nc -U
/var/run/clamd.scan/clamd.sock)
dummy-dd-10MB-with-zero.iso: OK
real 0m4.056s
user 0m0.008s
sys 0m0.004s

$ time (echo "SCAN dummy-dd-10MB-with-urandom.iso" | nc -U
/var/run/clamd.scan/clamd.sock)
dummy-dd-10MB-with-urandom.iso: OK
real 0m0.569s
user 0m0.012s
sys 0m0.000s
```


A 250MB file with /dev/zero: Takes almost 8 times as long as /dev/urandom.
```
$ dd if=/dev/zero of=dummy-dd-250MB-with-zero.iso bs=25MB count=10
$ dd if=/dev/urandom of=dummy-dd-250MB-with-urandom.iso bs=25MB count=10
$ ls -ltr dummy-dd-250MB-with-*
-rw-r--r-- 1 user user 250000000 Jul  9 03:44 dummy-dd-250MB-with-urandom.iso
-rw-r--r-- 1 user user 250000000 Jul  9 03:44 dummy-dd-250MB-with-zero.iso

$ time (echo "SCAN dummy-dd-250MB-with-zero.iso" | nc -U
/var/run/clamd.scan/clamd.sock)
dummy-dd-250MB-with-zero.iso: OK
real 1m42.949s
user 0m0.009s
sys 0m0.003s

$time (echo "SCAN dummy-dd-250MB-with-urandom.iso" | nc -U
/var/run/clamd.scan/clamd.sock)
dummy-dd-250MB-with-urandom.iso: OK
real 0m12.905s
user 0m0.004s
sys 0m0.007s
```

Thanks.

--
Taizo Ito

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml