[clamav-users] ClamAV mirrors have gotten worse!

[Paul Kosinski]

I believe that the delays we have been observing are due to some
problem with the Boston Cloudflare servers, or, perhaps, Comcast has a
"transparent" caching proxy which is causing us trouble.

I recently installed the same build and configuration of ClamAV 0.100.2
on our Web server, a virtual machine hosted in NYC. It runs the same
extra code (curl etc.) to check the cvd version number that we have
locally. Since Friday, there have been no delays there, although there
have been several significant delays locally. They check at exactly
the same time as each other (i.e., via NTP synced cron jobs).

I also am now running, at each location, simple curls to read the first
few bytes of the cvd files (to get the version number), *and* to log
all the headers sent and received. These are also run at exactly the
same time (as each other) via cron.

The headers show that our local system uses the 'BOS' Cloudflare server,
while the remote one uses the 'IAD' server:

  CF-RAY: 47fd0b7af79dae32-BOS
  CF-RAY: 47fd0b8064d9c1b8-IAD

Interestingly, these two cron jobs sometimes show that the BOS server
is out of date relative to the IAD server. For example, the following
curls show that one cvd file served by the BOS server is one version
behind that served by the IAD server at the *same* time. The files'
"Last-modified" lines are of particular interest. The BOS server says
the file was last modified on Mon, 26 Nov 2018 at 06:19:22 GMT, while
the IAD server says the file was last modified on Mon, 26 Nov 2018 at
14:15:24 GMT. 

In particular, the BOS "Date:" header says it's already about 14 mins
*later* than the IAD "Last-modified:" timestamp indicates. In other
words, the file delivered by the BOS server is, at time of *delivery*,
already about 14 minutes out of date.

--- BOS server ---

    ------------------------------  Monday 26 November 2018 at 09:29:01  ------------------------------

    /usr/bin/curl -4 -0 -s -v -r 0-99  -H Connection:close  -A ClamAV/0.100.2    database.clamav.net/daily.cvd  2>&1
    * About to connect() to database.clamav.net port 80 (#0)
    *   Trying 104.16.187.138...
    * connected
    * Connected to database.clamav.net (104.16.187.138) port 80 (#0)
    > GET /daily.cvd HTTP/1.0
    > Range: bytes=0-99
    > User-Agent: ClamAV/0.100.2
    > Host: database.clamav.net
    > Accept: */*
    > Connection:close
    >
    * additional stuff not fine transfer.c:1042: 0 0
    * HTTP 1.1 or later with persistent connection, pipelining supported
    < HTTP/1.1 206 Partial Content
    < Date: Mon, 26 Nov 2018 14:29:01 GMT
    < Content-Type: application/octet-stream
    < Content-Length: 100
    < Connection: close
    < Set-Cookie: __cfduid=d6eb82c36b149e2a4d07b430117606e581543242541; expires=Tue, 26-Nov-19 14:29:01 GMT; path=/; domain=.clamav.net; HttpOnly
    < Last-Modified: Mon, 26 Nov 2018 06:19:22 GMT
    < ETag: "5bfb906a-321a420"
    < Expires: Mon, 26 Nov 2018 18:23:39 GMT
    < Cache-Control: public, max-age=14078
    < CF-Cache-Status: HIT
    < Content-Range: bytes 0-99/52536352
    < Server: cloudflare
    < CF-RAY: 47fd0b7af79dae32-BOS
    <
    { [data not shown]
    * Closing connection #0
    ClamAV-VDB:26 Nov 2018 01-15 -0500:25154:2160594:63:083b8d4aa3824a865ac0e3ebeb3f7ce0:MT9OksyAAA34bbI

    ------------------------------  Monday 26 November 2018 at 09:29:01  ------------------------------


--- IAD server ---

    ------------------------------  Monday 26 November 2018 at 09:29:02  ------------------------------

    /usr/bin/curl -4 -0 -s -v -r 0-99  -H Connection:close  -A ClamAV/0.100.2    database.clamav.net/daily.cvd  2>&1
    * About to connect() to database.clamav.net port 80 (#0)
    *   Trying 104.16.185.138...
    * Connected to database.clamav.net (104.16.185.138) port 80 (#0)
    > GET /daily.cvd HTTP/1.0
    > Range: bytes=0-99
    > User-Agent: ClamAV/0.100.2
    > Host: database.clamav.net
    > Accept: */*
    > Connection:close
    >
    < HTTP/1.1 206 Partial Content
    < Date: Mon, 26 Nov 2018 14:29:02 GMT
    < Content-Type: application/octet-stream
    < Content-Length: 100
    < Connection: close
    < Set-Cookie: __cfduid=d426fd78ff1d6c6e42029baf939e5bbee1543242542; expires=Tue, 26-Nov-19 14:29:02 GMT; path=/; domain=.clamav.net; HttpOnly
    < Last-Modified: Mon, 26 Nov 2018 14:15:24 GMT
    < ETag: "5bfbfffc-321bb54"
    < Expires: Mon, 26 Nov 2018 18:23:46 GMT
    < Cache-Control: public, max-age=14084
    < CF-Cache-Status: HIT
    < Content-Range: bytes 0-99/52542292
    < Server: cloudflare
    < CF-RAY: 47fd0b8064d9c1b8-IAD
    <
    { [data not shown]
    * Closing connection 0
    ClamAV-VDB:26 Nov 2018 09-14 -0500:25155:2160841:63:9817036334370e1482f3fc58c6ed745a:MDvX2VW3tQr3ba4

    ------------------------------  Monday 26 November 2018 at 09:29:02  ------------------------------


P.S. As far as I can tell, there are no Cloudflare "blocks".


===================================================================================

On Tue, 20 Nov 2018 22:39:44 +0000
"Joel Esler (jesler)" <jesler at cisco.com> wrote:

> It's possible.  But, unless there is a vocal minority that no one is
> chiming in about, you are the only person/group that I have heard
> complain about the issue...
> 
> Millions of people are getting updates from Cloudflare a day, so
> something is working correctly, and there's been no configuration
> changes on our side.
> 
> If you receive Cloudflare blocks then that's a different story.