[clamav-users] Problem with private mirror and cld, inc files

[Kris Deugau]

Vangelis Katsikaros via clamav-users wrote:
> Hi Joel, thanks for the quick response. We already download once every 
> hour (the default ubuntu 18.04 behavior). However, we are using auto 
> scaling and we might be running a large number of EC2 instances (a few 
> hundreds), that could try to download simultaneously. To further scale 
> without issues it seemed quite easy (and it was easy in practice) to use 
> a private mirror. The system conditions and the cld & inc files is the 
> only blocker to make this work :/

It sounds like you need to add a freshclam call as one of the last bits 
of spinning up new VM/container instances just before actually 
activating one.  [edit:  I saw downthread you've supposedly already done 
this;  clearly something is going wrong close by that action.]  Or 
include a reference set of signature files and let freshclam run its 
updates in the VM/container as normally configured (with PrivateMirror, 
you should be able to get away with setting freshclam's update check 
interval down to 10-15 minutes - or possibly less).


>>     On Jan 26, 2021, at 11:46 AM, Vangelis Katsikaros via clamav-users
>>     <clamav-users at lists.clamav.net
>>     <mailto:clamav-users at lists.clamav.net>> wrote:
>>
>>     Hi
>>
>>     I am using Ubuntu 18.04 and recently found out that downloaind
>>     clamav files was blocked by cloudflare. Based on
>>     https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html
>>     I contacted Joel Esler (many thanks for the quick reply) and
>>     inform me that we were downloading too often.

>>     I did the following steps:
>>     1) Created a private mirror as described in
>>     https://www.clamav.net/documents/private-local-mirrors and
>>     mirrored all "cvd" files.
>>
>>     2) Cloud-init ensures that on boot the files are downloaded from
>>     the private mirror and are always available locally. Tthen
>>     cloud-init restarts the clamav daemon.

Double-check this step.  If the systemd condition is failing, or clamd 
is failing to find the files, then the files aren't getting where they 
need to.

>>     The systemd unit file has 2 ConditionPathExistsGlob that require
>>     the additional files "main.cld", "main.inc", "daily.cld",
>>     "daily.inc" to be available, for the service to start.
>>     $ cat /lib/systemd/system/clamav-daemon.service
>>     [Unit]
>>     Description=Clam AntiVirus userspace daemon
>>     Documentation=man:clamd(8) man:clamd.conf(5)
>>     https://www.clamav.net/documents/
>>     # Check for database existence
>>     ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
>>     ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}

Unless I misread, that means "at least one of main.cvd, main.cld, or 
main.inc, plus at least one of daily.cvd, daily.cld, or daily.inc";  you 
should not need more.  If you're using the stock signature files you'll 
usually see bytecode.(cvd|cld|inc) as well, but it's not considered 
critical.

I'm not sure about the .inc files (sounds a bit familiar, legacy 
historic files?).  The .cvd files are the compressed version of the .cld 
files - for long-running systems using the standard incremental updates 
you'll usually end up with .cld files, but I expect in your case you'll 
have .cvd files.  IIRC at one time you did actually end up with both - 
or all three - but checking several instances locally I only see the 
.cld files.

TBH, re-reading https://www.clamav.net/documents/private-local-mirrors 
it seems to me you'd be better off including a baseline set of .cvd or 
.cld files in your instance image (periodically updated to cut down the 
time for an instance to finish going live), and using option 1 - using a 
proxy server to let freshclam just do its incremental updates on your 
VPS/container instances without hitting the public mirror network each time.

-kgd