[clamav-users] Problem with private mirror and cld, inc files
Joel Esler (jesler)
jesler at cisco.com
Wed Jan 27 14:59:54 UTC 2021
As long as you are using fresh clam, which handles incremental delta updates, and the instances are running correctly (not stuck in a loop or something silly), the best place to get updates is from database.clamav.net <http://database.clamav.net/>
> On Jan 27, 2021, at 5:45 AM, Vangelis Katsikaros via clamav-users <clamav-users at lists.clamav.net> wrote:
>
> Thanks for the actions Joel, my IPs are now unblocked.
>
> I'd be interested to continue the discussion on
> a) how to find the main & daily .inc and .cld files
> b) why in ubuntu 18.04 there is a systemd condition for the clamav-daemon to run, but the files are not available for download
> I am afraid that in the long run my service could again DoS database.clamav.net <http://database.clamav.net/> and it would be prudent for me to proceed with the private mirror solution.
>
> Regards
> Vangelis
>
> On Tue, Jan 26, 2021 at 7:20 PM Joel Esler (jesler) <jesler at cisco.com <mailto:jesler at cisco.com>> wrote:
> Let me remove the block for your IP and see what we see.
>
>> On Jan 26, 2021, at 12:17 PM, Vangelis Katsikaros <vkatsikaros at gmail.com <mailto:vkatsikaros at gmail.com>> wrote:
>>
>> Hi Joel, thanks for the quick response. We already download once every hour (the default ubuntu 18.04 behavior). However, we are using auto scaling and we might be running a large number of EC2 instances (a few hundreds), that could try to download simultaneously. To further scale without issues it seemed quite easy (and it was easy in practice) to use a private mirror. The system conditions and the cld & inc files is the only blocker to make this work :/
>>
>> On Tue, Jan 26, 2021 at 7:00 PM Joel Esler (jesler) <jesler at cisco.com <mailto:jesler at cisco.com>> wrote:
>> Why wouldn’t you just download less often? Instead of doing all of this? We publish updates once a day, there’s no need to download the entire package once a second.
>>
>> --
>> Joel Esler
>> Manager, Communities Division
>> Cisco Talos Intelligence Group
>> http://www.talosintelligence.com <http://www.talosintelligence.com/> | https://www.snort.org <https://www.snort.org/>
>>
>>> On Jan 26, 2021, at 11:46 AM, Vangelis Katsikaros via clamav-users <clamav-users at lists.clamav.net <mailto:clamav-users at lists.clamav.net>> wrote:
>>>
>>> Hi
>>>
>>> I am using Ubuntu 18.04 and recently found out that downloaind clamav files was blocked by cloudflare. Based on https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html <https://lists.clamav.net/pipermail/clamav-users/2020-April/009482.html> I contacted Joel Esler (many thanks for the quick reply) and inform me that we were downloading too often.
>>>
>>> I tried to fix the situation by introducing a private mirror, but I ran into problems with cld and inc files and systemd conditions for the clamav-daemon service. I'd like to ask what are the "cld" and "inc" files, and how to download them in order to meet the systemd condition that requires them to be on disk?
>>>
>>> I did the following steps:
>>> 1) Created a private mirror as described in https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors> and mirrored all "cvd" files.
>>>
>>> 2) Cloud-init ensures that on boot the files are downloaded from the private mirror and are always available locally. Tthen cloud-init restarts the clamav daemon.
>>>
>>> 3) Configured freshclam with
>>> PrivateMirror https://my-clamav-mirror/clamav <https://my-clamav-mirror/clamav>
>>> ScriptedUpdates no
>>> Checks 3
>>>
>>> 4) Now freshclam works fine, however the clamav daemon does not, even if the cvd are available locally.
>>> $ sudo systemctl status clamav-daemon.service
>>> ● clamav-daemon.service - Clam AntiVirus userspace daemon
>>> Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled; vendor preset: enabled)
>>> Drop-In: /etc/systemd/system/clamav-daemon.service.d
>>> └─extend.conf
>>> Active: inactive (dead)
>>> Condition: start condition failed at Tue 2021-01-26 12:46:17 UTC; 3h 9min ago
>>> └─ ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc} was not met
>>> Docs: man:clamd(8)
>>> man:clamd.conf(5)
>>> https://www.clamav.net/documents/ <https://www.clamav.net/documents/>
>>>
>>> The systemd unit file has 2 ConditionPathExistsGlob that require the additional files "main.cld", "main.inc", "daily.cld", "daily.inc" to be available, for the service to start.
>>> $ cat /lib/systemd/system/clamav-daemon.service
>>> [Unit]
>>> Description=Clam AntiVirus userspace daemon
>>> Documentation=man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ <https://www.clamav.net/documents/>
>>> # Check for database existence
>>> ConditionPathExistsGlob=/var/lib/clamav/main.{c[vl]d,inc}
>>> ConditionPathExistsGlob=/var/lib/clamav/daily.{c[vl]d,inc}
>>>
>>> [Service]
>>> ExecStart=/usr/sbin/clamd --foreground=true
>>> # Reload the database
>>> ExecReload=/bin/kill -USR2 $MAINPID
>>> StandardOutput=syslog
>>> TimeoutStartSec=420
>>>
>>> [Install]
>>> WantedBy=multi-user.target
>>>
>>>
>>> 5) The "main.cld", "main.inc", "daily.cld", "daily.inc" files are not available to download from the main download site
>>>
>>> $ curl --head -s https://database.clamav.net/daily.cld <https://database.clamav.net/daily.cld> | grep HTTP
>>> HTTP/2 404
>>> $ curl --head -s https://database.clamav.net/daily.inc <https://database.clamav.net/daily.inc> | grep HTTP
>>> HTTP/2 404
>>> $ curl --head -s https://database.clamav.net/main.cld <https://database.clamav.net/main.cld> | grep HTTP
>>> HTTP/2 404
>>> $ curl --head -s https://database.clamav.net/main.inc <https://database.clamav.net/main.inc> | grep HTTP
>>> HTTP/2 404
>>>
>>>
>>> 6) I ran the suggested clamdownloader.pl <http://clamdownloader.pl/> (again from https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors>) but I don't see any
>>> $ ls -al /tmp/clam
>>> total 222848
>>> drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 26 16:27 .
>>> drwxrwxrwt 93 root root 12288 Jan 26 16:27 ..
>>> -rw-rw-r-- 1 ubuntu ubuntu 296388 Sep 19 2019 bytecode.cvd
>>> -rw-rw-r-- 1 ubuntu ubuntu 110005306 Jan 25 12:28 daily.cvd
>>> -rw-rw-r-- 1 ubuntu ubuntu 42 Jan 26 16:27 dns.txt
>>> -rw-rw-r-- 1 ubuntu ubuntu 117859675 Nov 25 2019 main.cvd
>>> drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 26 16:27 temp
>>>
>>> but I don't see any "main.cld", "main.inc", "daily.cld", "daily.inc" files.
>>>
>>> Repeating the question in the beginning: I'd like to ask what are the "cld" and "inc" files, and how to download in my private mirror in order to meet the systemd condition that requires them to be on disk?
>>>
>>> Regards
>>> Vangelis Katsikaros
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users at lists.clamav.net <mailto:clamav-users at lists.clamav.net>
>>> https://lists.clamav.net/mailman/listinfo/clamav-users <https://lists.clamav.net/mailman/listinfo/clamav-users>
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq <https://github.com/vrtadmin/clamav-faq>
>>>
>>> http://www.clamav.net/contact.html#ml <http://www.clamav.net/contact.html#ml>
>>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20210127/0c785dc9/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3010 bytes
Desc: not available
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20210127/0c785dc9/attachment.bin>
More information about the clamav-users
mailing list