[clamav-users] Update on rate limits and downloading

[Micah Snyder (micasnyd)]

Hi Mark,

I'm not sure how you got a config with the default set to "ReceiveTimeout 30".  I just tested with ubunte 20.04 a moment ago and a fresh `apt install clamav` (0.103.2).  The config I found in /etc/clamav/freshclam.conf has "ReceiveTimeout 0" which means it is disabled. ClamAV's built-in default (if you don't specify) is also "0". So I'm not really sure what went wrong for you.

So we do have a minor problem with the mirrors.dat in 0.103.2.  It will be owned by root instead of by the "clamav" user if you run "sudo freshclam --daemon". Then if you try running freshclam a different way, you may run into permissions issues.  We'll have to fix this in the next patch version ☹. 

-Micah
 
> -----Original Message-----
> From: clamav-users <clamav-users-bounces at lists.clamav.net> On Behalf Of
> clamav.mbourne at spamgourmet.com
> Sent: Thursday, May 6, 2021 1:12 PM
> To: clamav-users at lists.clamav.net
> Subject: Re: [clamav-users] Update on rate limits and downloading
> 
> Joel Esler jesler via clamav-users wrote:
> > Overall — we’re doing much better.
> >
> > We’ve reduced the amount of bandwidth we’re serving by 4x, so we’ve
> > made significant progress.
> >
> > /However, /we still have over 700 individual systems downloading the
> > full daily.cvd over 200x a day. (This should be once a day, /if
> > that/.)
> >
> > If you are not using 0.103.2 and it’s accompanying FreshClam to
> > download these updates, and when you do create a NEW FreshClam.conf
> > file and move your settings to that.  We’re going to have to start
> > blocking these atrocious abusers, as the rate limits are hurting
> > everyone else at this point.
> 
> I'm new to installing ClamAV, so there may be something I haven't done
> quite right here.  A couple of weeks ago, I installed ClamAV 0.103.2 from the
> Ubuntu repositories (clamav, clamav-freshclam, clamav-daemon, clamav-
> docs, clamtk and libclamunrar9 packages).
> 
> By default, FreshClam seems to use too short a download timeout and retry
> too frequently, triggering the rate limiting.  After installing, the FreshClam
> service would repeatedly attempt to download the daily.cvd file, time out
> after 30 seconds, and wait 5 seconds before trying again.
> After a few attempts, it then gets blocked by the CDN (if that's what "you are
> on cool-down" in the log means?) for 4 hours.  By the time I'd realised this
> was happening following the initial install, I was already blocked.
> 
> Perhaps this might, if left in a default configuration, be seen to attempt to
> download daily.cvd over 100 times a day, but without ever actually getting
> the whole file.  From what I'd seen here and in documentation / FAQs, I
> thought FreshClam was supposed to avoid retrying so frequently that it
> triggers the rate limiting?
> 
> I don't know if the default configuration is provided by ClamAV or the
> Ubuntu packaging (either way, it seems FreshClam shouldn't just keep
> retrying so quickly?)  In my case, freshclam.conf originally had
> "ReceiveTimeout 30".  Increasing it to 60 wasn't enough.  I then went to 600,
> which was successful.  Somewhere in between would probably have been
> fine, but incrementing more gradually would have been a long process,
> having to wait at least 4 hours between attempts (particularly as restarting
> FreshClam after setting a new timeout seems to get blocked for a further 4
> hours - not just the remainder of the original block).
> 
> In case it's of any use (and if this list allows it), I've attached my freshclam.log
> from those initial attempts.
> 
> All seems to be working OK now, but posting here in case the information is
> useful.
> 
> > Please help us, stay diligent, keep going keep upgrading.  Upgrade to
> > 0.103.2, and keep your mirrors.dat file around, this file contains a
> > snapshot of where you are in your update progression so that the next
> > time that FreshClam run, it can start where it left off.
> 
> Interesting you should mention mirrors.dat... Aside from the downloads
> timing out, there are also some errors in my freshclam.log about not being
> able to create mirrors.dat.  That's a bit odd, since the /var/lib/clamav/
> directory is owned and writeable by the correct user, but the mirrors.dat file
> within it is owned by root.  Deleting that file and restarting the freshclam
> service, the mirrors.dat file gets recreated, again owned by root.  That error
> hasn't appeared in the logs since, although mirrors.dat is still dated 25th April,
> so I'm not sure if there's still a problem with that.
> 
> --
> Mark.