[clamav-users] Problem with private mirror and cld, inc files

Tue Jan 26 17:17:57 UTC 2021

Hi Joel, thanks for the quick response. We already download once every hour
(the default ubuntu 18.04 behavior). However, we are using auto scaling and
we might be running a large number of EC2 instances (a few hundreds), that
could try to download simultaneously. To further scale without issues it
seemed quite easy (and it was easy in practice) to use a private mirror.
The system conditions and the cld & inc files is the only blocker to make
this work :/

On Tue, Jan 26, 2021 at 7:00 PM Joel Esler (jesler) <jesler at cisco.com>
wrote:

> Why wouldn’t you just download less often?  Instead of doing all of this?
> We publish updates once a day, there’s no need to download the entire
> package once a second.
>
> --
> Joel Esler
> Manager, Communities Division
> Cisco Talos Intelligence Group
> http://www.talosintelligence.com | https://www.snort.org
>
> On Jan 26, 2021, at 11:46 AM, Vangelis Katsikaros via clamav-users <
> clamav-users at lists.clamav.net> wrote:
>
> Hi
>
> I am using Ubuntu 18.04 and recently found out that downloaind clamav
> files was blocked by cloudflare. Based on
> https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html I
> contacted Joel Esler (many thanks for the quick reply) and inform me that
> we were downloading too often.
>
> I tried to fix the situation by introducing a private mirror, but I ran
> into problems with cld and inc files and systemd conditions for the
> clamav-daemon service. I'd like to ask what are the "cld" and "inc" files,
> and how to download them in order to meet the systemd condition that
> requires them to be on disk?
>
> I did the following steps:
> 1) Created a private mirror as described in
> https://www.clamav.net/documents/private-local-mirrors and mirrored all
> "cvd" files.
>
> 2) Cloud-init ensures that on boot the files are downloaded from the
> private mirror and are always available locally. Tthen cloud-init restarts
> the clamav daemon.
>
> 3) Configured freshclam with
> PrivateMirror https://my-clamav-mirror/clamav
> ScriptedUpdates no
> Checks 3
>
> 4) Now freshclam works fine, however the clamav daemon does not, even if
> the cvd are available locally.
> $ sudo systemctl status clamav-daemon.service
> ● clamav-daemon.service - Clam AntiVirus userspace daemon
>    Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled;
> vendor preset: enabled)
>   Drop-In: /etc/systemd/system/clamav-daemon.service.d
>            └─extend.conf
>    Active: inactive (dead)
> Condition: start condition failed at Tue 2021-01-26 12:46:17 UTC; 3h 9min
> ago
>            └─ ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
> was not met
>      Docs: man:clamd(8)
>            man:clamd.conf(5)
>            https://www.clamav.net/documents/
>
> The systemd unit file has 2 ConditionPathExistsGlob that require the
> additional files "main.cld", "main.inc", "daily.cld", "daily.inc" to be
> available, for the service to start.
> $ cat /lib/systemd/system/clamav-daemon.service
> [Unit]
> Description=Clam AntiVirus userspace daemon
> Documentation=man:clamd(8) man:clamd.conf(5)
> https://www.clamav.net/documents/
> # Check for database existence
> ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
> ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
>
> [Service]
> ExecStart=/usr/sbin/clamd --foreground=true
> # Reload the database
> ExecReload=/bin/kill -USR2 $MAINPID
> StandardOutput=syslog
> TimeoutStartSec=420
>
> [Install]
> WantedBy=multi-user.target
>
>
> 5) The "main.cld", "main.inc", "daily.cld", "daily.inc" files are not
> available to download from the main download site
>
> $ curl --head -s https://database.clamav.net/daily.cld  | grep HTTP
> HTTP/2 404
> $ curl --head -s https://database.clamav.net/daily.inc  | grep HTTP
> HTTP/2 404
> $ curl --head -s https://database.clamav.net/main.cld   | grep HTTP
> HTTP/2 404
> $ curl --head -s https://database.clamav.net/main.inc   | grep HTTP
> HTTP/2 404
>
>
> 6) I ran the suggested clamdownloader.pl (again from
> https://www.clamav.net/documents/private-local-mirrors) but I don't see
> any
> $ ls -al /tmp/clam
> total 222848
> drwxrwxr-x  3 ubuntu ubuntu      4096 Jan 26 16:27 .
> drwxrwxrwt 93 root   root       12288 Jan 26 16:27 ..
> -rw-rw-r--  1 ubuntu ubuntu    296388 Sep 19  2019 bytecode.cvd
> -rw-rw-r--  1 ubuntu ubuntu 110005306 Jan 25 12:28 daily.cvd
> -rw-rw-r--  1 ubuntu ubuntu        42 Jan 26 16:27 dns.txt
> -rw-rw-r--  1 ubuntu ubuntu 117859675 Nov 25  2019 main.cvd
> drwxrwxr-x  2 ubuntu ubuntu      4096 Jan 26 16:27 temp
>
> but I don't see any "main.cld", "main.inc", "daily.cld", "daily.inc" files.
>
> Repeating the question in the beginning: I'd like to ask what are the
> "cld" and "inc" files, and how to download in my private mirror in order to
> meet the systemd condition that requires them to be on disk?
>
> Regards
> Vangelis Katsikaros
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20210126/5b1cbb2c/attachment.htm>