[clamav-users] Unable to download clamav cvd file using google cloud python function

Thu Mar 11 23:21:22 UTC 2021

Yup. and that’s why people are getting 429’s

Sent from my  iPhone

> On Mar 10, 2021, at 23:01, Paul Kosinski <clamav-users at iment.com> wrote:
> 
> "I can’t play wack-a-mole with single IPs or even whole ASNs."
> 
> Does Cloudflare have the iptables hashlimit filter (or the equivalent) available?
> 
> 
> 
>> On Wed, 10 Mar 2021 22:29:41 +0000
>> "Joel Esler \(jesler\) via clamav-users" <clamav-users at lists.clamav.net> wrote:
>> 
>> To give everyone a frame of reference. This is what a Cdiff release and download cycle should look like:
>> 
>> 
>> [cid:311D041A-A699-48A6-BB74-8523A3927866]
>> 
>> Big influx right in the morning when we publish, and then peaks on the top and bottom of the hour every hour throughout a 24 hour period, (people having a cron job that runs at the top of every hour throughout the day) Theoretically speaking, at the end of 24 hours, the line should go to zero, it never will, because of new installs that download a bunch of cdiffs right in a row and things like that.  But I I look between the peaks find people like this:
>> 
>> [cid:B0884332-310A-4C6F-9960-A0A8DB6C2B0D]
>> 
>> 100 CDIFFs or so behind, and they download it nearly 2k times in a row?  Why?  This is not a partial download either.  It’s the full file.  Stuck cron?
>> 
>> Or this single IP:
>> 
>> [cid:AE797960-535D-44D1-AB4F-7C5823B5BBF2]
>> 
>> Who in the past 24 hours has created 22.17M file downloads all by themselves from a single IP. (The main.cvd btw)
>> 
>> It’s these bad apples that have ruined the basket for everyone.  I can’t play wack-a-mole with single IPs or even whole ASNs.
>> 
>> Multiply this one IP above x thousands, and you see the volume I am dealing with.  But that graph at the top there is from yesterday, and it’s much better.  This is what we are aiming for.  We’ve reduced transferred data by 60% by cutting back on abusers.
>> 
>> Like I said, I’ll be writing a blog post about this, but just to show you guys what I am dealing with:
>> 
>> [cid:D66E6145-0352-45EA-8579-5353C85C15F1]
>> 
>> In the past 72 hours, this is what our event graphs look like.  Big drop offs and increases are attributed to the constant adjustment I am doing to find the right balance.
>> 
>> --
>> Joel Esler
>> Manager, Communities Division
>> Cisco Talos Intelligence Group
>> http://www.talosintelligence.com | https://www.snort.org
>> 
>> On Mar 10, 2021, at 3:30 PM, Joel Esler (jesler) via clamav-users <clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>> wrote:
>> 
>> 
>> 
>> On Mar 10, 2021, at 12:31 PM, Paul Smith via clamav-users <clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>> wrote:
>> 
>> On 10/03/2021 17:00, Paul Kosinski via clamav-users wrote:
>> I wonder how many "ordinary" users of ClamAV are giving up on using it after getting permanent 403s. I would imagine there are lots of people who don't pursue the issue. They may even tell others that ClamAV is unreliable (which would tarnish its reputation).
>> 
>> Indeed. There does seem to be a view from some people here that anyone using ClamAV should be regularly updating, monitoring this list, monitoring blogs, etc. Ordinary people just don't do that.
>> 
>> I expect many will just be thinking that the database servers are broken, and are waiting for them to recover on their own (as they've done in the past) and they'll eventually go elsewhere.
>> 
>> The change should really be published everywhere possible - at least in big letters on the ClamAV home page, and possibly including going to popular computer press, etc.
>> 
>> A blog article (which is actually very hard to find) or announcement list post (which is even harder to find) which vaguely says that databases won't be tested on older versions isn't quite the same as a home page announcement that old versions & wget just won't work any more!
>> 
>> Of course, people have limited rights to complain - it's not like we're paying for it.
>> 
>> We are going to be writing a couple blog posts in the coming days.  I haven’t had the time to sit down and do it.
>> 