[clamav-users] Update on rate limits and downloading

[clamav.mbourne at spamgourmet.com]

Joel Esler jesler via clamav-users wrote:
> Overall — we’re doing much better.
> 
> We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve made 
> significant progress.
> 
> /However, /we still have over 700 individual systems downloading the 
> full daily.cvd over 200x a day. (This should be once a day, /if that/.)
> 
> If you are not using 0.103.2 and it’s accompanying FreshClam to download 
> these updates, and when you do create a NEW FreshClam.conf file and move 
> your settings to that.  We’re going to have to start blocking these 
> atrocious abusers, as the rate limits are hurting everyone else at this 
> point.

I'm new to installing ClamAV, so there may be something I haven't done 
quite right here.  A couple of weeks ago, I installed ClamAV 0.103.2 
from the Ubuntu repositories (clamav, clamav-freshclam, clamav-daemon, 
clamav-docs, clamtk and libclamunrar9 packages).

By default, FreshClam seems to use too short a download timeout and 
retry too frequently, triggering the rate limiting.  After installing, 
the FreshClam service would repeatedly attempt to download the daily.cvd 
file, time out after 30 seconds, and wait 5 seconds before trying again. 
After a few attempts, it then gets blocked by the CDN (if that's what 
"you are on cool-down" in the log means?) for 4 hours.  By the time I'd 
realised this was happening following the initial install, I was already 
blocked.

Perhaps this might, if left in a default configuration, be seen to 
attempt to download daily.cvd over 100 times a day, but without ever 
actually getting the whole file.  From what I'd seen here and in 
documentation / FAQs, I thought FreshClam was supposed to avoid retrying 
so frequently that it triggers the rate limiting?

I don't know if the default configuration is provided by ClamAV or the 
Ubuntu packaging (either way, it seems FreshClam shouldn't just keep 
retrying so quickly?)  In my case, freshclam.conf originally had 
"ReceiveTimeout 30".  Increasing it to 60 wasn't enough.  I then went to 
600, which was successful.  Somewhere in between would probably have 
been fine, but incrementing more gradually would have been a long 
process, having to wait at least 4 hours between attempts (particularly 
as restarting FreshClam after setting a new timeout seems to get blocked 
for a further 4 hours - not just the remainder of the original block).

In case it's of any use (and if this list allows it), I've attached my 
freshclam.log from those initial attempts.

All seems to be working OK now, but posting here in case the information 
is useful.

> Please help us, stay diligent, keep going keep upgrading.  Upgrade to 
> 0.103.2, and keep your mirrors.dat file around, this file contains a 
> snapshot of where you are in your update progression so that the next 
> time that FreshClam run, it can start where it left off.

Interesting you should mention mirrors.dat... Aside from the downloads 
timing out, there are also some errors in my freshclam.log about not 
being able to create mirrors.dat.  That's a bit odd, since the 
/var/lib/clamav/ directory is owned and writeable by the correct user, 
but the mirrors.dat file within it is owned by root.  Deleting that file 
and restarting the freshclam service, the mirrors.dat file gets 
recreated, again owned by root.  That error hasn't appeared in the logs 
since, although mirrors.dat is still dated 25th April, so I'm not sure 
if there's still a problem with that.

-- 
Mark.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freshclam.log
Type: text/x-log
Size: 36508 bytes
Desc: not available
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20210506/e083328d/attachment.bin>