[clamav-users] error code 429
clamav.mbourne at spamgourmet.com
clamav.mbourne at spamgourmet.com
Sun Sep 5 16:15:26 UTC 2021
No problem; good to know it was useful.
In my case, only the one host behind the NAT (physical PC on a home
broadband connection) is running freshclam anyway, but it appears I was
still being blocked by the rate-limiting. As I understand it, that
shouldn't usually have happened even with the per-IP system. Not sure
if that's an issue with how the new system differentiates between hosts,
or perhaps when the download failed (for whatever reason) freshclam was
trying several times and getting blocked.
I'm running Linux Mint 20, which is based on Ubuntu 20.04 and uses a lot
of packages from the Ubuntu repositories (upgraded not long after my
posts here a few months ago when I had problems with the default receive
timeout in Ubuntu 16/18.04's packages). ClamAV and freshclam are
installed from the Ubuntu 20.04 repositories, and I haven't yet needed
to change the configuration from the default - so my config will be the
same as anyone else who's installed from the Ubuntu 20.04 repo will have
by default. Not sure whether the new system would have treated everyone
with this default config as the same host, though I'd have thought IP
would still be taken into account as well.
I'm not complaining - you've clearly had a lot of problems with the CDN
being abused (intentionally or otherwise) and need to try these things.
Just trying to give you whatever information might be useful :)
Thanks,
Mark.
Joel Esler jesler via clamav-users - clamav-users at lists.clamav.net wrote:
> This is useful. Thank you.
>
> Each host should have a different rate limit under the new system (I turned it back off last night, which is why everyone got everything).
>
> Right now, the rate limit is “per IP”. So, if you have several
> Hosts behind a NAT, so you’ll get blocked. The new system, you can have as many hosts behind the same NAT as long as they aren’t using the same config file.
>
> A new problem being, I am seeing a ton of hosts on Amazon or Microsoft’s azure that are using the same config, so that’s a new hurdle that those people will have to overcome. I am sure there are new problems that we’ll encounter during this transition.
>
>
>
> —
> Sent from my iPhone
>
>> On Sep 5, 2021, at 09:09, clamav.mbourne at spamgourmet.com wrote:
>>
>> Joel Esler clamav-users at lists.clamav.net wrote:
>>> We are experimenting with a feature that we’ve been working with Cloudflare on, trying to isolate violators on a per host basis for the newest versions of ClamAV, instead of IP.
>>
>> I'm guessing you probably already have all the info you need but, in case it happens to be any help, this is what I have in my freshclam logs (on a home desktop PC, so it's not running 24-7)...
>>
>> Last messages from Friday:
>>> Fri Sep 3 22:13:18 2021 -> Received signal: wake up
>>> Fri Sep 3 22:13:18 2021 -> ClamAV update process started at Fri Sep 3 22:13:18 2021
>>> Fri Sep 3 22:13:18 2021 -> WARNING: Your ClamAV installation is OUTDATED!
>>> Fri Sep 3 22:13:18 2021 -> WARNING: Local version: 0.103.2 Recommended version: 0.103.3
>>> Fri Sep 3 22:13:18 2021 -> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
>>> Fri Sep 3 22:13:18 2021 -> daily.cld database is up-to-date (version: 26283, sigs: 1970262, f-level: 90, builder: ray
>>> nman)
>>> Fri Sep 3 22:13:18 2021 -> main.cvd database is up-to-date (version: 61, sigs: 6607162, f-level: 90, builder: sigmgr)
>>> Fri Sep 3 22:13:18 2021 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia
>>> 2)
>>> Fri Sep 3 22:13:18 2021 -> --------------------------------------
>>> Fri Sep 3 23:06:44 2021 -> Update process terminated
>>
>> So all was up-to-date then. Version 0.103.2 is the latest in the Ubuntu 20.04 repositories, which is why I'm on that version, hence the warning.
>>
>> First messages from Saturday:
>>> Sat Sep 4 11:54:21 2021 -> --------------------------------------
>>> Sat Sep 4 11:54:21 2021 -> freshclam daemon 0.103.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
>>> Sat Sep 4 11:54:21 2021 -> ClamAV update process started at Sat Sep 4 11:54:21 2021
>>> Sat Sep 4 11:54:21 2021 -> WARNING: Your ClamAV installation is OUTDATED!
>>> Sat Sep 4 11:54:21 2021 -> WARNING: Local version: 0.103.2 Recommended version: 0.103.3
>>> Sat Sep 4 11:54:21 2021 -> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
>>> Sat Sep 4 11:54:21 2021 -> daily database available for update (local version: 26283, remote version: 26284)
>>> Sat Sep 4 11:54:23 2021 -> WARNING: downloadPatch: Can't download daily-26284.cdiff from https://database.clamav.net/daily-26284.cdiff
>>> Sat Sep 4 11:54:23 2021 -> The database server doesn't have the latest patch for the daily database (version 26284). The server will likely have updated if you check again in a few hours.
>>> Sat Sep 4 11:54:23 2021 -> main.cvd database is up-to-date (version: 61, sigs: 6607162, f-level: 90, builder: sigmgr)
>>> Sat Sep 4 11:54:23 2021 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
>>> Sat Sep 4 11:54:23 2021 -> --------------------------------------
>>> Sat Sep 4 12:54:23 2021 -> Received signal: wake up
>>> Sat Sep 4 12:54:23 2021 -> ClamAV update process started at Sat Sep 4 12:54:23 2021
>>> Sat Sep 4 12:54:23 2021 -> WARNING: Your ClamAV installation is OUTDATED!
>>> Sat Sep 4 12:54:23 2021 -> WARNING: Local version: 0.103.2 Recommended version: 0.103.3
>>> Sat Sep 4 12:54:23 2021 -> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
>>> Sat Sep 4 12:54:23 2021 -> WARNING: FreshClam previously received error code 429 from the ClamAV Content Delivery Network (CDN).
>>> Sat Sep 4 12:54:23 2021 -> This means that you have been rate limited by the CDN.
>>> Sat Sep 4 12:54:23 2021 -> 1. Run FreshClam no more than once an hour to check for updates.
>>> Sat Sep 4 12:54:23 2021 -> FreshClam should check DNS first to see if an update is needed.
>>> Sat Sep 4 12:54:23 2021 -> 2. If you have more than 10 hosts on your network attempting to download,
>>> Sat Sep 4 12:54:23 2021 -> it is recommended that you set up a private mirror on your network using
>>> Sat Sep 4 12:54:23 2021 -> cvdupdate (https://pypi.org/project/cvdupdate/) to save bandwidth on the
>>> Sat Sep 4 12:54:23 2021 -> CDN and your own network.
>>> Sat Sep 4 12:54:23 2021 -> 3. Please do not open a ticket asking for an exemption from the rate limit,
>>> Sat Sep 4 12:54:23 2021 -> it will not be granted.
>>> Sat Sep 4 12:54:23 2021 -> WARNING: You are still on cool-down until after: 2021-09-04 15:54:23
>>
>> So at 11:54 it determined that an update was available but it couldn't be downloaded. It next checked an hour later at 12:54, and was apparently already rate-limited by then (for 2 checks an hour apart, after none for 12 hours). That was repeated at 13:43 and 14:54, then at 15:54:
>>> Sat Sep 4 15:54:23 2021 -> Received signal: wake up
>>> Sat Sep 4 15:54:23 2021 -> ClamAV update process started at Sat Sep 4 15:54:23 2021
>>> Sat Sep 4 15:54:23 2021 -> WARNING: Your ClamAV installation is OUTDATED!
>>> Sat Sep 4 15:54:23 2021 -> WARNING: Local version: 0.103.2 Recommended version: 0.103.3
>>> Sat Sep 4 15:54:23 2021 -> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
>>> Sat Sep 4 15:54:23 2021 -> WARNING: Cool-down expired, ok to try again.
>>> Sat Sep 4 15:54:23 2021 -> ERROR: Can't create mirrors.dat in /var/lib/clamav
>>> Sat Sep 4 15:54:23 2021 -> Hint: The database directory must be writable for UID XXX or GID YYY
>>> Sat Sep 4 15:54:23 2021 -> daily database available for update (local version: 26283, remote version: 26284)
>>> Sat Sep 4 15:54:24 2021 -> WARNING: downloadPatch: Can't download daily-26284.cdiff from https://database.clamav.net/daily-26284.cdiff
>>> Sat Sep 4 15:54:24 2021 -> The database server doesn't have the latest patch for the daily database (version 26284). The server will likely have updated if you check again in a few hours.
>>> Sat Sep 4 15:54:24 2021 -> main.cvd database is up-to-date (version: 61, sigs: 6607162, f-level: 90, builder: sigmgr)
>>> Sat Sep 4 15:54:24 2021 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
>>> Sat Sep 4 15:54:24 2021 -> --------------------------------------
>>
>> At 16:54, 17:54 and 18:54 it was back to "FreshClam previously received error code 429... you have been rate limited by the CDN". At 19:54 the cool-down expired and it was able to check again - but failed again the same as above. Then on cool-down at 20:54, 21:54 and 22:54, after which the PC was shut down. This is the only instance of freshclam running on my home network, and nothing else should be attempting to download the ClamAV databases (I haven't been trying to download them manually, or running other instances of freshclam).
>>
>> Today:
>>> Sun Sep 5 11:27:13 2021 -> --------------------------------------
>>> Sun Sep 5 11:27:13 2021 -> freshclam daemon 0.103.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
>>> Sun Sep 5 11:27:13 2021 -> ClamAV update process started at Sun Sep 5 11:27:13 2021
>>> Sun Sep 5 11:27:13 2021 -> WARNING: Your ClamAV installation is OUTDATED!
>>> Sun Sep 5 11:27:13 2021 -> WARNING: Local version: 0.103.2 Recommended version: 0.103.3
>>> Sun Sep 5 11:27:13 2021 -> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
>>> Sun Sep 5 11:27:13 2021 -> daily database available for update (local version: 26283, remote version: 26285)
>>> Sun Sep 5 11:27:15 2021 -> Testing database: '/var/lib/clamav/tmp.a9599a4ff7/clamav-431aa03fce17054479c616a2f44eae7b.tmp-daily.cld' ...
>>> Sun Sep 5 11:27:20 2021 -> Database test passed.
>>> Sun Sep 5 11:27:22 2021 -> daily.cld updated (version: 26285, sigs: 1970840, f-level: 90, builder: raynman)
>>> Sun Sep 5 11:27:22 2021 -> main.cvd database is up-to-date (version: 61, sigs: 6607162, f-level: 90, builder: sigmgr)
>>> Sun Sep 5 11:27:22 2021 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
>>> Sun Sep 5 11:27:22 2021 -> WARNING: Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.ctl: No such file or directory
>>> Sun Sep 5 11:27:22 2021 -> --------------------------------------
>>> Sun Sep 5 12:27:23 2021 -> Received signal: wake up
>>> Sun Sep 5 12:27:23 2021 -> ClamAV update process started at Sun Sep 5 12:27:23 2021
>>> Sun Sep 5 12:27:23 2021 -> WARNING: Your ClamAV installation is OUTDATED!
>>> Sun Sep 5 12:27:23 2021 -> WARNING: Local version: 0.103.2 Recommended version: 0.103.3
>>> Sun Sep 5 12:27:23 2021 -> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
>>> Sun Sep 5 12:27:23 2021 -> daily.cld database is up-to-date (version: 26285, sigs: 1970840, f-level: 90, builder: raynman)
>>> Sun Sep 5 12:27:23 2021 -> main.cvd database is up-to-date (version: 61, sigs: 6607162, f-level: 90, builder: sigmgr)
>>> Sun Sep 5 12:27:23 2021 -> bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
>>> Sun Sep 5 12:27:23 2021 -> --------------------------------------
>>
>> So it was able to successfully update today.
>>
>> --
>> Mark.
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users at lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
More information about the clamav-users
mailing list