[clamav-users] ClamAV® blog: Freshclam, cdiffs and bandwidth are your friends

[Joel Esler (jesler)]

Feel free to check the TXT record once an hour or whenever you want.  Checking the TXT record will tell you if there is a diff to download, for sure, and then you can go download that diff.

The problem isn’t that, the problem is downloading the ENTIRE main.cvd and daily.cvd once a minute, every minute (or in some cases, several times in the same minute)

Sent from my  iPad

> On Jul 28, 2020, at 19:02, Paul Kosinski via clamav-users <clamav-users at lists.clamav.net> wrote:
> 
> "...we also only release updates once a day."
> 
> Are there *never* any urgent virus updates released in between? In
> other words, is it always useless to check the TXT record more often?
> 
> 
> 
>> On Mon, 27 Jul 2020 22:09:31 +0000
>> "Joel Esler \(jesler\) via clamav-users" <clamav-users at lists.clamav.net> wrote:
>> 
>> https://blog.clamav.net/2020/07/freshclam-cdiffs-effect-on-bandwidth.html<https://blog.clamav.net/2020/07/freshclam-cdiffs-effect-on-bandwidth.html?m=1>
>> 
>> Freshclam, cdiffs and bandwidth are your friends
>> During a recent review of file downloads from our ClamAV CDN network, we've noticed hundreds of IPs that seem to be downloading the daily.cvd and the main.cvd thousands of times a day.
>> 
>> There are about a dozen IPs that are downloading those to files more than 40,000 times a day. This is causing us to transfer about 250TB of data a day. We would encourage any users still doing this to cease as soon as possible. Not only does it waste our bandwidth — as we have much more efficient ways of downloading the updates — but it also wastes your bandwidth, as well.
>> 
>> Freshclam has the ability to download partial files of updates (called cdiffs).  Which are smaller, more incremental updates to the database. This allows users, and us, to manage our downloads in a much more efficient manner. We often receive the complaint, "I have to download the daily.cvd and main.cvd with Python and move the updates to an off-internet system."  That's fine — it's a use case we support. However, you can do the same with freshclam and the small cdiffs.
>> 
>> Furthermore, we also only release updates once a day.  Reducing the number of updates you check for (and, subsequently, download we assume through a crontab or periodic job of some type) would also alleviate this issue.
>> 
>> We will be constantly monitoring this in hopes that people migrate to using freshclam.  Over-abusers (for instance, the top 10 IPs that are downloading main.cvd 40,000 times a day), will be immediately blocked.  Further abusers may also be blocked, without notice.
>> 
>> To mitigate, please complete the following tasks:
>> 
>> 1. Use Freshclam instead of Python or whatever downloading script you have cron'd.
>> 2. Reduce the checks to once or twice a day.
>> 
>> Thank you for helping keep the ClamAV network healthy.
>> 
>> Any questions, please see us over on the ClamAV-Users list.
>> 
>> 
>> Sent from my  iPhone
> 
> _______________________________________________
> 
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
> 
> 
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
> 
> http://www.clamav.net/contact.html#ml
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1872 bytes
Desc: not available
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20200729/0aa47604/attachment.bin>