This is getting rather technical, and probably some of CloudFlare’s secret sauce.
It sounds like the anycast DNS that cloudflare hosts isn’t really working, or at least I would assume that they are using anycast.

So you query current.cvd.clamav.net but are getting different results at IAD and BOS.  Now next is the inclusion of Comcast, which may and probably is caching DNS records beyond normal TTLs which could cause the difference.  I personally always run an Unbound cache server on my mailserver networks to cache dns for at least an hour for rbls that I’m not rsyncing, but that could cause an issue with Microsoft’s wonderful 10 second MX records.  So that’s where I’ve run into this issue, but not often enough since I’m just caching for an hour and probably MS expects it.

So my guess, is probably not anycast, but a caching DNS server that is still giving older records.

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300

On Dec 7, 2018, at 6:20 PM, Paul Kosinski <clamav-users@iment.com> wrote:

As some of you may be aware, ever since ClamAV began using Cloudflare,
we have seen many occasions when files like daily.cvd were not
available to our LAN until well after the DNS TXT record implied they
should be.

However, we discovered that these same files *are* available to our
Web/email server right away. So what is the difference? The first
difference is that our Web server (a VM) is offsite, and is served by
the "IAD" Cloudflare complex, whereas our local setup is served by the
"BOS" Cloudflare complex.

The second, and likely explanatory difference, is that our local setup
is connected via Comcast (a dynamic IP and all that), while our Web
server (with its static IP etc.) is almost certainly more directly
connected to the Internet as a whole.

The workaround we have adopted is as follows: we installed a "tinyproxy"
server on our offsite VM. To ensure it only proxys for us, it listens on
the encrypted OpenVPN tunnel we already had in place for FTP uploads
etc. Then, instead of directly accessing database.clamav.net, freshclam
uses our remote VM as a proxy,so that the cvd files are downloaded
indirectly from Cloudflare's IAD server complex (via tinyproxy) rather
than directly from Cloudflare's BOS server complex.

Since switching to this workaround a few days ago, we haven't observed
any delays: the cvd files are available right away when the DNS TXT
query says they should be.

I strongly suspect that Comcast is the culprit in the delays that had
plagued us. This is especially suggested by the fact that Cloudflare
returns a "Cache-Control:" header similar to:

 Cache-Control: public, max-age=13672

where the max-age value varies, but is often several hours.

In my opinion, for data like ClamAV virus updates, the "Cache-Control:"
should specify "no-cache". Can Cloudflare do this for ClamAV?

---------------------------------------------------------------------

Below is a pair of recent (pre-workaround) log excerpts. They show a
delay of over 2.5 hours experienced from BOS (via Comcast) vs no delay
from IAD.

Note that the BOS "Date:" timestamp of 16:49:01 GMT *still* shows
a "Last-Modified:" timestamp of 06:15:18 GMT, while IAD already shows
the up-to-date "Last-Modified:" timestamp of 14:14:30 GMT at the much
earlier "Date:" of 14:29:01 GMT!


 IAD

   Date: Sun, 02 Dec 2018 14:09:01 GMT
   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
   ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT

   Date: Sun, 02 Dec 2018 14:29:01 GMT
   Last-Modified: Sun, 02 Dec 2018 14:14:30 GMT
   ClamAV-VDB:02 Dec 2018 09-13 -0500:25173:2167842:63:ba557f61737b9d4b66acc96f7044b524:3nBAOxo97ssSNZb


 BOS

   Date: Sun, 02 Dec 2018 14:09:01 GMT
   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
   ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT

   Date: Sun, 02 Dec 2018 14:29:01 GMT
   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
   ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT

   Date: Sun, 02 Dec 2018 14:49:01 GMT
   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
   ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT

   Date: Sun, 02 Dec 2018 15:09:01 GMT
   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
   ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT

   Date: Sun, 02 Dec 2018 15:29:02 GMT
   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
   ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT

   Date: Sun, 02 Dec 2018 15:49:02 GMT
   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
   ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT

   Date: Sun, 02 Dec 2018 16:09:01 GMT
   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
   ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT

   Date: Sun, 02 Dec 2018 16:29:01 GMT
   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
   ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT

   Date: Sun, 02 Dec 2018 16:49:01 GMT
   Last-Modified: Sun, 02 Dec 2018 06:15:18 GMT
   ClamAV-VDB:02 Dec 2018 01-14 -0500:25172:2167574:63:13c670e3a525c4fd17bf65524ff05fcd:nwPmlNwUbKmexgT

   Date: Sun, 02 Dec 2018 17:09:01 GMT
   Last-Modified: Sun, 02 Dec 2018 14:14:30 GMT
   ClamAV-VDB:02 Dec 2018 09-13 -0500:25173:2167842:63:ba557f61737b9d4b66acc96f7044b524:3nBAOxo97ssSNZb
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml