[clamav-users] ClamAV signature update sync errors have gotten worse

[Paul Kosinski]

It's good to save so much (5 PB) Internet traffic.

What we were seeing from our end was that there were a lot of full-size
downloads of daily.cvd that were useless because they were the old
version rather than the new version advertised by the DNS TXT record.

Besides being annoying because of lots of extra logging by freshclam,
it kept killing off the mirror IP addresses due to the update failures,
and thus eventually blocked all downloads.

Since we already had a wrapper around freshclam to do some extra stuff
in our environment, I decided to write the extra code to only invoke
freshclam if the prefix of the cvd file(s) showed the correct version.
After that, it was easy to log the delay to separate file.

I guess my question at this point is: how many other users of freshclam
are seeing the problem we had? The behavior we were seeing not only
wasted bandwidth, it also caused semi-permanent blockage of future
updates. Users who don't monitor their logs (like many desktop users?)
could be far out of date with their ClamAV signatures.

P.S. It shouldn't be too hard to modify freshclam itself to deal with
this problem in a similar fashion. But I didn't want to fork a fairly
complicated program which mainly does stuff that has nothing to do with
this particular problem.



On Mon, 20 Aug 2018 15:43:14 +0000
"Joel Esler (jesler)" <jesler at cisco.com> wrote:

> Thank you.  We have to make adjustments very slowly to not disrupt
> anyone.
> 
> Cloudflare has helped us save 2 PB in the last month, delivering
> updates an average of 39% faster.  We are seeing excellent results.  
> 
> > On Aug 18, 2018, at 1:09 AM, Paul Kosinski <clamav-users at iment.com>
> > wrote:
> > 
> > Joel,
> > 
> > Still lots of delays since "2018-08-11 13:18:02  No delay", but none
> > quite as long as the previous batch:
> > 
> > 2018-08-11 21:33:02  00:15:00 delay
> > 2018-08-12 05:48:02  01:00:00 delay
> > 2018-08-12 14:33:01  01:15:00 delay
> > 2018-08-12 22:48:02  01:00:00 delay
> > 2018-08-13 05:18:01  No delay
> > 2018-08-13 13:18:02  No delay
> > 2018-08-13 21:33:01  00:14:59 delay
> > 2018-08-14 05:18:01  No delay
> > 2018-08-14 13:18:02  No delay
> > 2018-08-14 21:33:02  00:30:01 delay
> > 2018-08-15 05:03:02  No delay
> > 2018-08-15 13:48:01  00:45:00 delay
> > 2018-08-15 22:03:01  No delay
> > 2018-08-16 05:03:02  No delay
> > 2018-08-16 14:03:02  01:00:01 delay
> > 2018-08-16 21:18:01  00:14:59 delay
> > 2018-08-17 06:03:01  No delay
> > 2018-08-17 13:33:02  00:30:01 delay
> > 2018-08-17 21:03:02  No delay
> > 
> > 
> > On Thu, 16 Aug 2018 22:13:48 +0000
> > "Joel Esler (jesler)" <jesler at cisco.com> wrote:
> > 
> >> Paul, how are things looking from your side?

>