[clamav-users] Detected duplicate databases

Al Varnell alvarnell at mac.com
Thu Jan 18 21:45:49 UTC 2018


As I mentioned before, I don't use a private mirror, so I've dropped out of the conversation. Someone who does use option 2 needs to double-check your settings here.

-Al-

On Thu, Jan 18, 2018 at 10:07 AM, Hugo Deprez wrote:
> Hello,
> 
> I don't think I'm the only one having this issue. It seems that I used the
> official configuration from http://www.clamav.net/doc/mirrors-faq.html <http://www.clamav.net/doc/mirrors-faq.html>
> Should I report the issue somewhere ?
> 
> The only way I see to avoid this issue is to create a shell script and to
> push the updates my self on all servers, but this is not the proper way to
> do this...
> 
> Hugo
> 
> On 15 January 2018 at 11:44, Hugo Deprez <hugo.deprez at gmail.com <mailto:hugo.deprez at gmail.com>> wrote:
> 
>> Hello,
>> 
>> by using this freshclam.conf  :
>> 
>> AllowSupplementaryGroups false
>> Bytecode true
>> Checks 24
>> CompressLocalDatabase no
>> ConnectTimeout 30
>> DNSDatabaseInfo current.cvd.clamav.net <http://current.cvd.clamav.net/>
>> DatabaseDirectory /var/lib/clamav
>> DatabaseMirror mirror.recette.local
>> DatabaseOwner clamav
>> Debug false
>> Foreground false
>> LogFacility LOG_LOCAL6
>> LogFileMaxSize 0
>> LogRotate true
>> LogSyslog true
>> LogTime yes
>> LogVerbose false
>> MaxAttempts 5
>> PidFile /var/run/clamav/freshclam.pid
>> #PrivateMirror mirror.recette.local
>> ReceiveTimeout 30
>> ScriptedUpdates no
>> TestDatabases yes
>> UpdateLogFile /var/log/clamav/freshclam.log
>> 
>> I got the old issue :
>> 
>> root at server:/var/lib/clamav# rm -f *
>> root at server:/var/lib/clamav# freshclam
>> ClamAV update process started at Mon Jan 15 11:38:48 2018
>> Downloading main.cvd [100%]
>> main.cvd updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
>> Downloading daily.cvd [100%]
>> daily.cvd updated (version: 24223, sigs: 1825336, f-level: 63, builder:
>> neo)
>> Downloading bytecode.cvd [100%]
>> bytecode.cvd updated (version: 315, sigs: 75, f-level: 63, builder:
>> raynman)
>> Database updated (6391660 signatures) from mirror.recette.local (IP:
>> 192.168.1.15)
>> root at raadmin:/var/lib/clamav# freshclam
>> ClamAV update process started at Mon Jan 15 11:40:00 2018
>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
>> sigmgr)
>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
>> builder: neo)
>> Downloading bytecode.cvd [100%]
>> WARNING: Mirror 192.168.8.15 is not synchronized.
>> Trying again in 5 secs...
>> ClamAV update process started at Mon Jan 15 11:40:05 2018
>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
>> sigmgr)
>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
>> builder: neo)
>> WARNING: Can't download bytecode.cvd from mirror.recette.local
>> Trying again in 5 secs...
>> ClamAV update process started at Mon Jan 15 11:40:10 2018
>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
>> sigmgr)
>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
>> builder: neo)
>> WARNING: Can't download bytecode.cvd from mirror.recette.local
>> Trying again in 5 secs...
>> ClamAV update process started at Mon Jan 15 11:40:15 2018
>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
>> sigmgr)
>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
>> builder: neo)
>> WARNING: Can't download bytecode.cvd from mirror.recette.local
>> Trying again in 5 secs...
>> ClamAV update process started at Mon Jan 15 11:40:20 2018
>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
>> sigmgr)
>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
>> builder: neo)
>> ERROR: Can't download bytecode.cvd from mirror.recette.local
>> Giving up on mirror.recette.local ...
>> Update failed. Your network may be down or none of the mirrors listed in
>> /etc/clamav/freshclam.conf is working. Check http://www.clamav.net/doc/ <http://www.clamav.net/doc/>
>> mirrors-faq.html for possible reasons.
>> 
>> 
>> 
>> On 14 January 2018 at 10:36, Al Varnell <alvarnell at mac.com <mailto:alvarnell at mac.com>> wrote:
>> 
>>> 
>>> On Sat, Jan 13, 2018 at 04:12 AM, Hugo Deprez wrote:
>>>> 
>>>> hello,
>>>> 
>>>> sorry I wasn't clear :
>>>> 
>>>> - I used solution number 3 or more than one year, but freshclam on
>>> clients
>>>> was randomly reporting error on downloading cvd files. I never found the
>>>> solution :
>>>> 
>>>> *Tue Dec  1 17:54:34 2015 -> WARNING: Can't download daily.cvd from *
>>>> 
>>>> *mirror*
>>>> 
>>>> - So now I switch to solution number 2 from
>>>> https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors>
>>>> 
>>>> My "/etc/clamav/freshclam.conf" is :
>>>> 
>>>> *AllowSupplementaryGroups falseBytecode trueChecks
>>>> 24CompressLocalDatabase noConnectTimeout 30DNSDatabaseInfo
>>>> current.cvd.clamav.net
>>>> <http://current.cvd.clamav.net>DatabaseDirectory
>>>> /var/lib/clamavDatabaseMirror mirror.recette.localDatabaseOwner
>>>> clamavDebug falseForeground falseLogFacility LOG_LOCAL6LogFileMaxSize
>>>> 0LogRotate trueLogSyslog trueLogTime yesLogVerbose falseMaxAttempts
>>>> 5PidFile /var/run/clamav/freshclam.pidPrivateMirror
>>>> mirror.recette.localReceiveTimeout 30ScriptedUpdates noTestDatabases
>>>> yesUpdateLogFile /var/log/clamav/freshclam.log*
>>>> 
>>>> But when I run freshclam, it still downloading the cld files :
>>> 
>>> Because you changed PrivateMirror to mirror.recette.local. See
>>> freshclam.conf:
>>> 
>>> # This option allows you to easily point freshclam to private mirrors.
>>> # If PrivateMirror is set, freshclam does not attempt to use DNS
>>> # to determine whether its databases are out-of-date, instead it will
>>> # use the If-Modified-Since request or directly check the headers of the
>>> # remote database files. For each database, freshclam first attempts
>>> # to download the CLD file. If that fails, it tries to download the
>>> # CVD file. This option overrides DatabaseMirror, DNSDatabaseInfo
>>> # and ScriptedUpdates. It can be used multiple times to provide
>>> # fall-back mirrors.
>>> # Default: disabled
>>> #PrivateMirror mirror1.mynetwork.com <http://mirror1.mynetwork.com/>
>>> 
>>> Try commenting that line out.
>>> 
>>>> *# freshclam*
>>>> 
>>>> *ClamAV update process started at Sat Jan 13 13:07:47 2018Reading CVD
>>>> header (main.cld): OK (IMS)Reading CVD header (main.cvd): OK
>>>> (IMS)main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60,
>>>> builder: sigmgr)Reading CVD header (daily.cld): OKDownloading
>>>> daily.cld [100%]daily.cld updated (version: 24217, sigs: 1823727,
>>>> f-level: 63, builder: neo)Reading CVD header (bytecode.cld):
>>>> OKbytecode.cvd is up to date (version: 319, sigs: 75, f-level: 63,
>>>> builder: neo)[LibClamAV] Detected duplicate databases
>>>> /var/lib/clamav/daily.cvd and /var/lib/clamav/daily.cld. The
>>>> /var/lib/clamav/daily.cvd database is older and will not be loaded,
>>>> you should manually remove it from the database directory.Database
>>>> updated (6390051 signatures) from mirror.recette.local (IP:
>>>> 192.168.1.15)*If I remove cld files from my private mirror :
>>>> 
>>>> *# freshclamClamAV update process started at Sat Jan 13 13:10:18
>>>> 2018Downloading main.cld [100%]main.cld updated (version: 58, sigs:
>>>> 4566249, f-level: 60, builder: sigmgr)Downloading daily.cld
>>>> [100%]daily.cld updated (version: 24217, sigs: 1823727, f-level: 63,
>>>> builder: neo)WARNING: getfile: bytecode.cld not found on
>>>> mirror.recette.local (IP: 192.168.1.15)WARNING: Can't download
>>>> bytecode.cld from mirror.recette.localDownloading bytecode.cvd
>>>> [100%]bytecode.cvd updated (version: 319, sigs: 75, f-level: 63,
>>>> builder: neo)Database updated (6390051 signatures) from
>>>> mirror.recette.local (IP: 192.168.1.15)*
>>>> 
>>>> I just want to run freshclam without any error.
>>>> 
>>>> * Any idea ? *
>>>> 
>>>> On 11 January 2018 at 10:37, Al Varnell <alvarnell at mac.com <mailto:alvarnell at mac.com>> wrote:
>>>> 
>>>>> On Thu, Jan 11, 2018 at 01:15 AM, Hugo Deprez wrote:
>>>>>> Hello,
>>>>>> 
>>>>>> thank you for the answer.
>>>>>> I don't think my freshclam is trying to download the cdiff file
>>> because I
>>>>>> use : ScriptedUpdates no
>>>>>> 
>>>>>> Is this parameter you are talking about incremental updates ?
>>>>> 
>>>>> Yes, that's what I was referring to.
>>>>> 
>>>>>> I know this is not bandwith efficient, but I had to many issues with
>>>>>> freshclam and my local repository made by clamdownloader.pl
>>>>> 
>>>>> So you are using an option 3 private local mirror as described in <
>>>>> https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors>>. Make sure you
>>>>> also change freshclam.conf "DatabaseMirror machine1.mylan" where
>>>>> machine1.mylan is the name of your mirror server.
>>>>> 
>>>>> Also make sure you are using the latest version of the
>>> clamavdownloader.pl
>>>>> <https://github.com/akissa/clamav-faq/blob/master/ <https://github.com/akissa/clamav-faq/blob/master/>
>>>>> mirrors/clamdownloader.pl>
>>>>> 
>>>>> The existing clamdownloader.pl script does not have any error
>>> correction
>>>>> it simply bails out if a downloaded file is not valid and is unable to
>>>>> retry different mirrors if one fails. That is the most likely reason
>>> for
>>>>> those 404 errors.
>>>>> 
>>>>> I have no experience with private local mirrors, but you might have
>>> better
>>>>> luck with the clamavmirror script <https://pypi.python.org/pypi/ <https://pypi.python.org/pypi/>
>>>>> clamavmirror/0.0.3>.
>>>>> 
>>>>> -Al-
>>>>> 
>>>>>> Best regards,
>>>>>> 
>>>>>> On 10 January 2018 at 10:20, Al Varnell <alvarnell at mac.com <mailto:alvarnell at mac.com> <mailto:
>>>>> alvarnell at mac.com <mailto:alvarnell at mac.com>>> wrote:
>>>>>> 
>>>>>>> The first time freshclam pulls down a daily.cdiff file, your
>>> daily.cvd
>>>>>>> file will be decompressed to daily.cld and the .cdiff file added to
>>> it.
>>>>>>> From that point on you should only have the daily.cld file. One
>>>>> exception
>>>>>>> is that if for some reason freshclam is unable to find needed .cdiff
>>>>> files
>>>>>>> on a mirror, it will download a new .cvd file which will start the
>>>>> process
>>>>>>> again.
>>>>>>> 
>>>>>>> It is possible to reconfigure freshclam to download .cvd files by
>>>>>>> disabling incremental updates, but that would be very inefficient
>>> use of
>>>>>>> bandwidth and mirror server time, so unless you have a overriding
>>> need
>>>>> for
>>>>>>> that, I don't recommend it.
>>>>>>> 
>>>>>>> -Al-
>>>>>>> 
>>>>>>> On Wed, Jan 10, 2018 at 01:12 AM, Hugo Deprez wrote:
>>>>>>>> Hello,
>>>>>>>> 
>>>>>>>> I have a question about daily.cvd and daily.cld files. If I
>>> understood
>>>>>>>> correctly, those two files are almost the same : one is compressed,
>>> the
>>>>>>>> other is not.
>>>>>>>> Still I have an issue in my setup :
>>>>>>>> 
>>>>>>>> If I put both filtes in apache2 server (which act as PrivateMirror)
>>> :
>>>>>>>> 
>>>>>>>> -rw-r--r-- 1 www-data www-data 117892267 07.06.2017 23:38 main.cvd
>>>>>>>> -rw-r--r-- 1 www-data www-data    153228 07.12.2017 03:17
>>> bytecode.cvd
>>>>>>>> lrwxrwxrwx 1 root     root             8 05.01.2018 10:14 main.cld
>>> ->
>>>>>>>> main.cvd
>>>>>>>> lrwxrwxrwx 1 root     root             9 05.01.2018 10:14 daily.cld
>>> ->
>>>>>>>> daily.cvd
>>>>>>>> lrwxrwxrwx 1 root     root            12 05.01.2018 10:14
>>> bytecode.cld
>>>>> ->
>>>>>>>> bytecode.cvd
>>>>>>>> -rw-r--r-- 1 www-data www-data  43804052 10.01.2018 06:17 daily.cvd
>>>>>>>> 
>>>>>>>> On the client side I have this Warning :
>>>>>>>> 
>>>>>>>> LibClamAV Warning: Detected duplicate databases
>>>>> /var/lib/clamav/daily.cvd
>>>>>>>> and /var/lib/clamav/daily.cld, please manually remove one of them
>>>>>>>> 
>>>>>>>> 
>>>>>>>> But If I remove the *.cld files on my PrivateMirror I got multiples
>>> 404
>>>>>>>> errors from the Freshclam clients.
>>>>>>>> 
>>>>>>>> Is there a way to configure Freshclam in order to grabe only the cvd
>>>>>>> files
>>>>>>>> ?
>>>>>>>> 
>>>>>>>> Best regards,



More information about the clamav-users mailing list