[clamav-users] PrivateMirror set on client machine. Disable cld downloads

Paul Kosinski clamav-users at iment.com
Tue Mar 24 09:57:48 EDT 2020


I made a lot of postings last year about our ultimately unsuccessful
attempts to continue using a private mirror after ClamAV switched to
Cloudflare.

In summary, I had set up a wrapper for freshclam that would
periodically use the DNS TXT record, followed by a curl of the first
~100 bytes of the CVDs, to determine when to download new CVDs to our
mirror server. Unfortunately, the CVD files on *our* local Cloudflare
server (BOS) were, as often as not, behind the CVD version numbers the
DNS TXT reported. (Other Cloudflare servers, like IAD, were better.)

But, since we only have a few machines running ClamAV, a local mirror
didn't really save bandwidth, so I gave up and just download the CDIFFs
to each machine. The downside to this is that I had to allow (strictly
limited) direct Internet access from a machine that previously had none.

I also tried using a simple HTTP proxy (in the IAD Anycast region) for
a while, but I suspect the new HTTPS stuff would make this impractical.



On Mon, 23 Mar 2020 20:57:16 +0000
"Scott A. Wozny via clamav-users" <clamav-users at lists.clamav.net> wrote:

> That's a very valid point.  I hesitated to mention it only because my
> experience with distributed mirrors has been that the files on each
> mirror are timestamped when the mirror you happen to be talking to at
> that moment downloaded the file which is inconsistent over the pool.
> I wondered if, perhaps, the clamav mirroring structure had found a
> way around that issue, but when I see freshclam does either a DNS TXT
> check or a "first 512 byte header comparison" check it made me think
> this was still an issue and if-modified-since couldn't be relied upon
> to work until you got to the absolute last mirror to pull down the
> absolute newest version which could still be pretty inefficient
> overall (depending on the size of the pool compared to how often
> files change).  AND this issue would percolate down to the clients of
> the PrivateMirror unless, of course, the issue with differing
> timestamps from mirror to mirror has been resolved in some way.
> 
> So, I'm curious if your wget --timestamping approach has been tested
> in this context as truly downloading each file only once or if your
> point was to note the capability of the wget command?
> 
> Thanks,
> 
> Scott
> 
> ________________________________
> From: clamav-users <clamav-users-bounces at lists.clamav.net> on behalf
> of Arjen de Korte via clamav-users <clamav-users at lists.clamav.net>
> Sent: March 23, 2020 4:35 PM To: ClamAV users ML
> <clamav-users at lists.clamav.net> Cc: Arjen de Korte
> <build+clamav at de-korte.org> Subject: Re: [clamav-users] PrivateMirror
> set on client machine. Disable cld downloads
> 
> Citeren "Scott A. Wozny via clamav-users"
> <clamav-users at lists.clamav.net>:
> 
> > One caveat with that suggestion is that if you move off of freshclam
> > to do your signature retrieval with wget, you give up the
> > efficiencies of just downloading the first 512 bytes of each DB file
> > to see if it's been updated and, if not, going back to sleep until
> > the next check.  Using wget you go from a few KB of bandwidth per
> > server per check to hundreds of MB for each update on each server.
> > If you don't care about bandwidth and disk I/O, then this difference
> > won't matter to you, but if it does, just be aware of the
> > difference.  
> 
> The --timestamping option of wget will fix that for you and will only
> download the file if the timestamp of the remote file changes:
> 
>      wget --timestamping http://database.clamav.net/daily.cvd
> 
> > Alternatively, you could emulate what freshclam does and curl the
> > first 512 bytes, do the comparative version checking and then
> > sleeping or doing the full download.  But once you get to that
> > point, you're practically rewriting freshclam and then the option of
> > reaching out to the devel list to modify freshclam to fit your needs
> > for a custom build starts to become a comparable amount of work.
> >
> > Scott
> >
> > ________________________________
> > From: clamav-users <clamav-users-bounces at lists.clamav.net> on behalf
> > of G.W. Haywood via clamav-users <clamav-users at lists.clamav.net>
> > Sent: March 23, 2020 12:32 PM
> > To: ClamAV users ML <clamav-users at lists.clamav.net>
> > Cc: G.W. Haywood <clamav at jubileegroup.co.uk>
> > Subject: Re: [clamav-users] PrivateMirror set on client machine.
> > Disable cld downloads
> >
> > Hi there,
> >
> > On Mon, 23 Mar 2020, vin9999 via clamav-users wrote:
> >  
> >> ... how can we disable .cld downloads? ...  
> >
> > Use something else to download the files?
> >
> > Presumably you use freshclam to keep the private mirror up to date,
> > but there is nothing which forces you to use freshclam to copy files
> > from the private mirror to its clients.  A utility like 'wget' could
> > do what you need.
> >
> > --
> >
> > 73,
> > Ged.


More information about the clamav-users mailing list