That's a very valid point.  I hesitated to mention it only because my experience with distributed mirrors has been that the files on each mirror are timestamped when the mirror you happen to be talking to at that moment downloaded the file which is inconsistent over the pool.  I wondered if, perhaps, the clamav mirroring structure had found a way around that issue, but when I see freshclam does either a DNS TXT check or a "first 512 byte header comparison" check it made me think this was still an issue and if-modified-since couldn't be relied upon to work until you got to the absolute last mirror to pull down the absolute newest version which could still be pretty inefficient overall (depending on the size of the pool compared to how often files change).  AND this issue would percolate down to the clients of the PrivateMirror unless, of course, the issue with differing timestamps from mirror to mirror has been resolved in some way.

So, I'm curious if your wget --timestamping approach has been tested in this context as truly downloading each file only once or if your point was to note the capability of the wget command?

Thanks,

Scott

From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of Arjen de Korte via clamav-users <clamav-users@lists.clamav.net>
Sent: March 23, 2020 4:35 PM
To: ClamAV users ML <clamav-users@lists.clamav.net>
Cc: Arjen de Korte <build+clamav@de-korte.org>
Subject: Re: [clamav-users] PrivateMirror set on client machine. Disable cld downloads
 
Citeren "Scott A. Wozny via clamav-users" <clamav-users@lists.clamav.net>:

> One caveat with that suggestion is that if you move off of freshclam 
> to do your signature retrieval with wget, you give up the 
> efficiencies of just downloading the first 512 bytes of each DB file 
> to see if it's been updated and, if not, going back to sleep until 
> the next check.  Using wget you go from a few KB of bandwidth per 
> server per check to hundreds of MB for each update on each server.  
> If you don't care about bandwidth and disk I/O, then this difference 
> won't matter to you, but if it does, just be aware of the difference.

The --timestamping option of wget will fix that for you and will only
download the file if the timestamp of the remote file changes:

     wget --timestamping http://database.clamav.net/daily.cvd

> Alternatively, you could emulate what freshclam does and curl the 
> first 512 bytes, do the comparative version checking and then 
> sleeping or doing the full download.  But once you get to that 
> point, you're practically rewriting freshclam and then the option of 
> reaching out to the devel list to modify freshclam to fit your needs 
> for a custom build starts to become a comparable amount of work.
>
> Scott
>
> ________________________________
> From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf 
> of G.W. Haywood via clamav-users <clamav-users@lists.clamav.net>
> Sent: March 23, 2020 12:32 PM
> To: ClamAV users ML <clamav-users@lists.clamav.net>
> Cc: G.W. Haywood <clamav@jubileegroup.co.uk>
> Subject: Re: [clamav-users] PrivateMirror set on client machine. 
> Disable cld downloads
>
> Hi there,
>
> On Mon, 23 Mar 2020, vin9999 via clamav-users wrote:
>
>> ... how can we disable .cld downloads? ...
>
> Use something else to download the files?
>
> Presumably you use freshclam to keep the private mirror up to date,
> but there is nothing which forces you to use freshclam to copy files
> from the private mirror to its clients.  A utility like 'wget' could
> do what you need.
>
> --
>
> 73,
> Ged.
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml




_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml