[clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing

[Joel Esler (jesler)]

Cloudflare's cache timeout is set to 5 seconds.  So, I would doubt that Cloudflare's cache is the issue, it may be an ISP thing in the middle doing the caching, which is what Paul is guessing at this point, if I am following the thread correctly.

Out of an abundance of caution I did a worldwide flush of daily.cvd yesterday.  Which caused everyone to get a new copy if it didn't match what they had.  This resulted in about 3TB of traffic in 10 minutes, but after that it settled back down.  We're still a bit higher than normal, as I eased some of the "you're going to fast" restrictions.  (I have a rate limiter set up, if you are downloading 100 cdiffs in 10 seconds, to rate limit the offender...)  I've disabled this for now

We're up to about 71TB a day right now and it seems to be holding steady.  Give it a couple more days and see if it comes back down.

--
Joel Esler
Manager, Communities Division
Cisco Talos Intelligence Group
http://www.talosintelligence.com

> On Dec 10, 2018, at 9:56 PM, Eric Tykwinski <eric-list at truenet.com> wrote:
> 
> Paul,
> 
> Sorry some of this confusion is probably my fault trying to help without going back to the whole thread.
> 
>> On Dec 10, 2018, at 9:34 PM, Paul Kosinski <clamav-users at iment.com> wrote:
>> 
>> We ARE using freshclam to perform the actual update. And always have
>> been!
>> 
>> We've only been using curl (not wget, if that matters) to pull the first
>> few bytes of the cvd to see if its version number matches what the DNS
>> TXT query said.
>> 
>> We do this because, after the conversion to Cloudflare, we were getting
>> lots of FAILURES where *freshclam* said things were out of sync (and
>> eventually disabled all the mirrors).
> 
> Have you tried what I did below?  I.E. curl/wget/telnet whatever your flavor of the day, and pull the newest cdiff?
> If you’re getting a 404, that’s definitely an issue.  
> 
> My guess is that it’s actually timing out though, and could be more of an issue troubleshooting.
> Is it local, ie an IDP getting stuck scanning the files, or remotely freshclam itself is timing out on BOS pulling the update from ClamAV and caching it before you can download it.
> 
>> And we have recently seen that our Web server sometimes can get the new
>> updates (from IAD) *hours* before our main LAN does (from BOS).
> 
> Those hours before are only checking the CVDs, which can and probably are cached on CloudFlare so not up to date.
> My guess is that there are just more people in Boston using Clam, so the cache last the longest.
> 
>> P.S. It's been quite frustrating getting some replies seemingly based on
>> assumptions that we are doing things we shouldn't, when we aren't in
>> fact doing those things. (Like not using freshclam.)
> 
> I would agree, this has gone on a long time from my recollection, which is why I jumped in and started looking at it.
> Definitely, I did hop on without all the facts and was just trying to figure out on the fly what’s going on, so my bad on that.
> 
> When in doubt, I usually pull a pcap on a server.  There’s a lot of factors that can come into play, but actually with clam only using http, this actually makes it a lot easier.
> 
> Sincerely,
> 
> Eric Tykwinski
> 
> 
> _______________________________________________
> clamav-users mailing list
> clamav-users at lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3010 bytes
Desc: not available
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20181212/fd520cbf/attachment.bin>