[clamav-users] A workaround for the major ClamAV DB update delays we have been experiencing
Joel Esler (jesler)
jesler at cisco.com
Tue Dec 11 21:23:55 EST 2018
Cloudflare's cache timeout is set to 5 seconds. So, I would doubt that Cloudflare's cache is the issue, it may be an ISP thing in the middle doing the caching, which is what Paul is guessing at this point, if I am following the thread correctly.
Out of an abundance of caution I did a worldwide flush of daily.cvd yesterday. Which caused everyone to get a new copy if it didn't match what they had. This resulted in about 3TB of traffic in 10 minutes, but after that it settled back down. We're still a bit higher than normal, as I eased some of the "you're going to fast" restrictions. (I have a rate limiter set up, if you are downloading 100 cdiffs in 10 seconds, to rate limit the offender...) I've disabled this for now
We're up to about 71TB a day right now and it seems to be holding steady. Give it a couple more days and see if it comes back down.
Manager, Communities Division
Cisco Talos Intelligence Group
> On Dec 10, 2018, at 9:56 PM, Eric Tykwinski <eric-list at truenet.com> wrote:
> Sorry some of this confusion is probably my fault trying to help without going back to the whole thread.
>> On Dec 10, 2018, at 9:34 PM, Paul Kosinski <clamav-users at iment.com> wrote:
>> We ARE using freshclam to perform the actual update. And always have
>> We've only been using curl (not wget, if that matters) to pull the first
>> few bytes of the cvd to see if its version number matches what the DNS
>> TXT query said.
>> We do this because, after the conversion to Cloudflare, we were getting
>> lots of FAILURES where *freshclam* said things were out of sync (and
>> eventually disabled all the mirrors).
> Have you tried what I did below? I.E. curl/wget/telnet whatever your flavor of the day, and pull the newest cdiff?
> If you’re getting a 404, that’s definitely an issue.
> My guess is that it’s actually timing out though, and could be more of an issue troubleshooting.
> Is it local, ie an IDP getting stuck scanning the files, or remotely freshclam itself is timing out on BOS pulling the update from ClamAV and caching it before you can download it.
>> And we have recently seen that our Web server sometimes can get the new
>> updates (from IAD) *hours* before our main LAN does (from BOS).
> Those hours before are only checking the CVDs, which can and probably are cached on CloudFlare so not up to date.
> My guess is that there are just more people in Boston using Clam, so the cache last the longest.
>> P.S. It's been quite frustrating getting some replies seemingly based on
>> assumptions that we are doing things we shouldn't, when we aren't in
>> fact doing those things. (Like not using freshclam.)
> I would agree, this has gone on a long time from my recollection, which is why I jumped in and started looking at it.
> Definitely, I did hop on without all the facts and was just trying to figure out on the fly what’s going on, so my bad on that.
> When in doubt, I usually pull a pcap on a server. There’s a lot of factors that can come into play, but actually with clam only using http, this actually makes it a lot easier.
> Eric Tykwinski
> clamav-users mailing list
> clamav-users at lists.clamav.net
> Help us build a comprehensive ClamAV guide:
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 3010 bytes
Desc: not available
More information about the clamav-users