[clamav-users] Slow database loading

[Ferdinand Goldmann]

Hello,

recently I discovered that my clamd db reload times have gone up
significantly. I am using a lot of 3rd party signatures from
clamav-unofficial-sigs.

Until last week, reloading took about 20 seconds:

Nov 13 00:14:01 mail3 clamd[82168]: Reading databases from /var/db/clamav
Nov 13 00:14:22 mail3 clamd[82168]: Database correctly reloaded (7922241 signatures)

A while later this went up to about 40 seconds:

Nov 13 20:14:01 mail3 clamd[82168]: Reading databases from /var/db/clamav
Nov 13 20:14:38 mail3 clamd[82168]: Database correctly reloaded (7948734 signatures)

Still later, load times are up to over one minute, which leads to
clients getting disconnected:

Nov 18 03:44:01 mail3 clamd[82168]: Reading databases from /var/db/clamav
Nov 18 03:45:14 mail3 clamd[82168]: Database correctly reloaded (7996388 signatures)
Nov 18 03:45:14 mail3 clamd[82168]: Client disconnected (FD 12)
Nov 18 03:45:14 mail3 clamd[82168]: Client disconnected (FD 11)
Nov 18 03:45:14 mail3 clamd[82168]: Client disconnected (FD 41)
Nov 18 03:45:14 mail3 clamd[82168]: Client disconnected (FD 43)

I made some adjustments like giving the clamd process higher priority
and providing my VMs with more CPU cores. This did help somewhat:
Nov 22 16:45:01 mail3 clamd[49508]: Reading databases from /var/db/clamav
Nov 22 16:46:01 mail3 clamd[49508]: Database correctly reloaded (8008037 signatures)

Does anyone else experience such high db reload times with when using the
unofficial signatures? It looks like the number of 3rd party signatures
grew a bit since last week. Also there are two fairly large ign2 files:

-rw-r--r--  1 spamd  spamd  6645 Nov 18 11:10 /var/db/clamav/securiteinfo.ign2
-rw-r--r--  1 spamd  spamd  6808 Nov 14 12:36 /var/db/clamav/sigwhitelist.ign2

When running clamscan --debug I noticed that the majority of time seems
to be spent reading/filtering the ignore list. I guess it might be related to
the way Clamav handles signatures internally, as described here:
https://bugzilla.clamav.net/show_bug.cgi?id=11017#c4

Any thoughts and input are appreciated!

Kind regards,
Ferdinand
-- 
Ferdinand Goldmann
System Administrator
Information Management

JOHANNES KEPLER
UNIVERSITY LINZ
Altenberger Straße 69
Hochschulfond Building, HF9902
4040 Linz, Austria
P +43 732 2468 3925
ferdinand.goldmann at jku.at
www.jku.at/im
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3900 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20161122/07f0a7cc/attachment.bin>