[clamav-users] still can't have ScriptedUpdates on a private mirror?

Mon May 5 18:02:09 UTC 2014

Has anyone been able to get the new PrivateMirror option to actually
work if the mirror is using ScriptedUpdates and thus has a mix of
*.cvd and *.cld files?

Here's the /etc/freshclam.conf file on our mirror:

$ awk -F# '{print $1}' /etc/freshclam.conf | grep -Ev '^[[:space:]]*$'
LogSyslog yes
LogFacility LOG_DAEMON
DatabaseMirror db.us.clamav.net
DatabaseMirror database.clamav.net
HTTPProxyServer proxy.example.org
HTTPProxyPort 8080
SafeBrowsing yes

Here's the /etc/freshclam.conf file on all of the clients:

$ awk -F# '{print $1}' /etc/freshclam.conf | grep -Ev '^[[:space:]]*$'
LogSyslog yes
LogFacility LOG_DAEMON
DatabaseMirror database.clamav.net
PrivateMirror clamav.example.org
SafeBrowsing yes

The man page states:

    This option allows you to easily point freshclam to private
    mirrors.  If PrivateMirror is set, freshclam does not attempt to
    use DNS to determine whether its databases are out-of-date,
    instead it will use the If-Modified-Since request or directly
    check the headers of the remote database files.  For each
    database, freshclam first attempts to download the CLD file.  If
    that fails, it tries to download the CVD file.  This option
    overrides DatabaseMirror, DNSDatabaseInfo and ScriptedUpdates.  It
    can be used multiple times to provide fallback mirrors.  Default:
    disabled

The problem we're seeing is that the clients assert that the daily.cld
file they download from the mirror is corrupted.

Here's the daily.cld info from the mirror:

$ sigtool --info daily.cld
File: daily.cld
Build time: 05 May 2014 11:13 -0400
Version: 18927
Signatures: 931802
Functionality level: 63
Builder: neo
Verification OK.

Here's the daily.cvd info on the client:

$ sigtool --info daily.cvd
File: daily.cvd
Build time: 02 May 2014 11:36 -0400
Version: 18908
Signatures: 923818
Functionality level: 63
Builder: neo
MD5: e8b57f86cd1351b14678af312b8c92ba
Digital signature: K91nZY+ONqzeaFH1P8nf6uJzDrcXWpAGZ1JUJHO42kaQ+e2w3Ud31d+enINHDWZANvByBu45e9boqxV2uswzoDycDa3dFSdThKyjCshnCCXJywOekiqoXl+/x45dV9Xtk/TCYvEpPDyPKvcoAamzysnasHvPXFGMFSz8OuWzcmf
Verification OK.

And here's what happens when the client runs freshclam:

$ ( umask 022 && freshclam --verbose )
...
Trying to retrieve CVD header of http://clamav.example.org/daily.cld
OK
Retrieving http://clamav.example.org/daily.cld
Trying to download http://clamav.example.org/daily.cld (IP: xxx.xxx.xxx.xxx)
Downloading daily.cld [100%]
ERROR: Verification: Can't verify database integrity
Retrieving http://clamav.example.org/daily.cvd
Trying to download http://clamav.example.org/daily.cvd (IP: xxx.xxx.xxx.xxx)
WARNING: getfile: daily.cvd not found on remote server (IP: xxx.xxx.xxx.xxx)
WARNING: Can't download daily.cvd from clamav.example.org

But if I manually download the file with wget, sigtool has no issues
with it:

$ cd /tmp
$ wget --quiet http://clamav.example.org/daily.cld
$ sigtool --info daily.cld
File: daily.cld
Build time: 05 May 2014 11:13 -0400
Version: 18927
Signatures: 931802
Functionality level: 63
Builder: neo
Verification OK.

I thought that freshclam might not be downloading daily.cld correctly.
So I ran freshclam, suspended it while it was downloading the
daily.cld file, and made a hard link to the temporary file so I could
inspect it after freshclam finished.

But the temporary file isn't corrupted; it has the exact same contents
as the daily.cld file on the mirror.

On the mirror:

$ sha256sum daily.cld
3739b2f09140df61d723e1efbde57bae3070acb0dc6c4c618b6eeaddd419e797  daily.cld

On the client:

$ sha256sum clamav-2b2cab7332cd4e2f99cfe7113f7488b1.tmp
3739b2f09140df61d723e1efbde57bae3070acb0dc6c4c618b6eeaddd419e797  clamav-2b2cab7332cd4e2f99cfe7113f7488b1.tmp

But on the client, sigtool throws the exact same error as freshclam
does:

$ sigtool --info clamav-2b2cab7332cd4e2f99cfe7113f7488b1.tmp
File: clamav-2b2cab7332cd4e2f99cfe7113f7488b1.tmp
Build time: 05 May 2014 11:13 -0400
Version: 18927
Signatures: 931802
Functionality level: 63
Builder: neo
ERROR: cvdinfo: Verification: Can't verify database integrity

>From experimenting, I determined that sigtool will correctly verify
the integrity of the temporary file, but if and only if the filename
ends with ".cld":

$ ln clamav-2b2cab7332cd4e2f99cfe7113f7488b1.tmp clamav-2b2cab7332cd4e2f99cfe7113f7488b1.cld
$ sigtool --info clamav-2b2cab7332cd4e2f99cfe7113f7488b1.cld
File: clamav-2b2cab7332cd4e2f99cfe7113f7488b1.cld
Build time: 05 May 2014 11:13 -0400
Version: 18927
Signatures: 931802
Functionality level: 63
Builder: neo
Verification OK.

>From looking at the source code, it would seem that sigtool really
does use the filename extension to determine the database type,
instead of just parsing the database.

This is extremely non-intuitive and highly undesirable behavior.  No
proper tool should make assumptions about the type of file it is
dealing with by its filename--particularly not a verification tool.

My guess is that the implementation of the PrivateMirror feature is
similarly buggy: the verification routines assume that the temporary
file is a CVD file (because PrivateMirror disables ScriptedUpdates),
and thus it erroneously fails to verify valid CLD files.  But logic
elsewhere in freshclam prefers CLD files to CVD files.

There is also further buggy logic with freshclam and PrivateMirror: if
PrivateMirror is enabled, and neither the CLD nor CVD file exists for
a database, freshclam won't even attempt to download the CLD file from
the mirror; it *only* attempts to download the CVD file.  This
directly contradicts what both the man page and the comments in the
sample freshclam.conf say:

    For each database, freshclam first attempts to download the CLD
    file.  If that fails, it tries to download the CVD file.

The only way I can think of to work around these bugs is to--once
again--go back to disabling ScriptedUpdates on the mirror, so the
mirror only ever has CVD files.  But this means our mirror is going to
be pulling the entire ~57 MiB daily.cvd file from db.us.clamav.net,
many times per day.

We're very grateful to all of the developers who work on Clam AV.
Overall, it serves our needs very well.  But it is a continual
frustration for us that freshclam functions so poorly with private
mirrors.

Yes, the PrivateMirror option is a step in the right direction.  But
the current implementation doesn't allow us to do what we've been
trying to do literally for years: have a private mirror that can
actually enable ScriptedUpdates, and thus not pound the crap out of
the main Clam AV mirrors on daily basis.