[clamav-users] Detected duplicate databases

Thomas McCourt (tmccourt) tmccourt at cisco.com
Thu Jan 18 22:15:34 UTC 2018


Hello,

Please enter a bug ticket in https://bugzilla.clamav.net/
For the private mirror issue.


I will look into it.

Thank you,


Tom M
On 1/18/18, 4:45 PM, "clamav-users on behalf of Al Varnell" <clamav-users-bounces at lists.clamav.net on behalf of alvarnell at mac.com> wrote:

>As I mentioned before, I don't use a private mirror, so I've dropped out of the conversation. Someone who does use option 2 needs to double-check your settings here.
>
>-Al-
>
>On Thu, Jan 18, 2018 at 10:07 AM, Hugo Deprez wrote:
>> Hello,
>> 
>> I don't think I'm the only one having this issue. It seems that I used the
>> official configuration from http://www.clamav.net/doc/mirrors-faq.html <http://www.clamav.net/doc/mirrors-faq.html>
>> Should I report the issue somewhere ?
>> 
>> The only way I see to avoid this issue is to create a shell script and to
>> push the updates my self on all servers, but this is not the proper way to
>> do this...
>> 
>> Hugo
>> 
>> On 15 January 2018 at 11:44, Hugo Deprez <hugo.deprez at gmail.com <mailto:hugo.deprez at gmail.com>> wrote:
>> 
>>> Hello,
>>> 
>>> by using this freshclam.conf  :
>>> 
>>> AllowSupplementaryGroups false
>>> Bytecode true
>>> Checks 24
>>> CompressLocalDatabase no
>>> ConnectTimeout 30
>>> DNSDatabaseInfo current.cvd.clamav.net <http://current.cvd.clamav.net/>
>>> DatabaseDirectory /var/lib/clamav
>>> DatabaseMirror mirror.recette.local
>>> DatabaseOwner clamav
>>> Debug false
>>> Foreground false
>>> LogFacility LOG_LOCAL6
>>> LogFileMaxSize 0
>>> LogRotate true
>>> LogSyslog true
>>> LogTime yes
>>> LogVerbose false
>>> MaxAttempts 5
>>> PidFile /var/run/clamav/freshclam.pid
>>> #PrivateMirror mirror.recette.local
>>> ReceiveTimeout 30
>>> ScriptedUpdates no
>>> TestDatabases yes
>>> UpdateLogFile /var/log/clamav/freshclam.log
>>> 
>>> I got the old issue :
>>> 
>>> root at server:/var/lib/clamav# rm -f *
>>> root at server:/var/lib/clamav# freshclam
>>> ClamAV update process started at Mon Jan 15 11:38:48 2018
>>> Downloading main.cvd [100%]
>>> main.cvd updated (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
>>> Downloading daily.cvd [100%]
>>> daily.cvd updated (version: 24223, sigs: 1825336, f-level: 63, builder:
>>> neo)
>>> Downloading bytecode.cvd [100%]
>>> bytecode.cvd updated (version: 315, sigs: 75, f-level: 63, builder:
>>> raynman)
>>> Database updated (6391660 signatures) from mirror.recette.local (IP:
>>> 192.168.1.15)
>>> root at raadmin:/var/lib/clamav# freshclam
>>> ClamAV update process started at Mon Jan 15 11:40:00 2018
>>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
>>> sigmgr)
>>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
>>> builder: neo)
>>> Downloading bytecode.cvd [100%]
>>> WARNING: Mirror 192.168.8.15 is not synchronized.
>>> Trying again in 5 secs...
>>> ClamAV update process started at Mon Jan 15 11:40:05 2018
>>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
>>> sigmgr)
>>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
>>> builder: neo)
>>> WARNING: Can't download bytecode.cvd from mirror.recette.local
>>> Trying again in 5 secs...
>>> ClamAV update process started at Mon Jan 15 11:40:10 2018
>>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
>>> sigmgr)
>>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
>>> builder: neo)
>>> WARNING: Can't download bytecode.cvd from mirror.recette.local
>>> Trying again in 5 secs...
>>> ClamAV update process started at Mon Jan 15 11:40:15 2018
>>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
>>> sigmgr)
>>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
>>> builder: neo)
>>> WARNING: Can't download bytecode.cvd from mirror.recette.local
>>> Trying again in 5 secs...
>>> ClamAV update process started at Mon Jan 15 11:40:20 2018
>>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder:
>>> sigmgr)
>>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
>>> builder: neo)
>>> ERROR: Can't download bytecode.cvd from mirror.recette.local
>>> Giving up on mirror.recette.local ...
>>> Update failed. Your network may be down or none of the mirrors listed in
>>> /etc/clamav/freshclam.conf is working. Check http://www.clamav.net/doc/ <http://www.clamav.net/doc/>
>>> mirrors-faq.html for possible reasons.
>>> 
>>> 
>>> 
>>> On 14 January 2018 at 10:36, Al Varnell <alvarnell at mac.com <mailto:alvarnell at mac.com>> wrote:
>>> 
>>>> 
>>>> On Sat, Jan 13, 2018 at 04:12 AM, Hugo Deprez wrote:
>>>>> 
>>>>> hello,
>>>>> 
>>>>> sorry I wasn't clear :
>>>>> 
>>>>> - I used solution number 3 or more than one year, but freshclam on
>>>> clients
>>>>> was randomly reporting error on downloading cvd files. I never found the
>>>>> solution :
>>>>> 
>>>>> *Tue Dec  1 17:54:34 2015 -> WARNING: Can't download daily.cvd from *
>>>>> 
>>>>> *mirror*
>>>>> 
>>>>> - So now I switch to solution number 2 from
>>>>> https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors>
>>>>> 
>>>>> My "/etc/clamav/freshclam.conf" is :
>>>>> 
>>>>> *AllowSupplementaryGroups falseBytecode trueChecks
>>>>> 24CompressLocalDatabase noConnectTimeout 30DNSDatabaseInfo
>>>>> current.cvd.clamav.net
>>>>> <http://current.cvd.clamav.net>DatabaseDirectory
>>>>> /var/lib/clamavDatabaseMirror mirror.recette.localDatabaseOwner
>>>>> clamavDebug falseForeground falseLogFacility LOG_LOCAL6LogFileMaxSize
>>>>> 0LogRotate trueLogSyslog trueLogTime yesLogVerbose falseMaxAttempts
>>>>> 5PidFile /var/run/clamav/freshclam.pidPrivateMirror
>>>>> mirror.recette.localReceiveTimeout 30ScriptedUpdates noTestDatabases
>>>>> yesUpdateLogFile /var/log/clamav/freshclam.log*
>>>>> 
>>>>> But when I run freshclam, it still downloading the cld files :
>>>> 
>>>> Because you changed PrivateMirror to mirror.recette.local. See
>>>> freshclam.conf:
>>>> 
>>>> # This option allows you to easily point freshclam to private mirrors.
>>>> # If PrivateMirror is set, freshclam does not attempt to use DNS
>>>> # to determine whether its databases are out-of-date, instead it will
>>>> # use the If-Modified-Since request or directly check the headers of the
>>>> # remote database files. For each database, freshclam first attempts
>>>> # to download the CLD file. If that fails, it tries to download the
>>>> # CVD file. This option overrides DatabaseMirror, DNSDatabaseInfo
>>>> # and ScriptedUpdates. It can be used multiple times to provide
>>>> # fall-back mirrors.
>>>> # Default: disabled
>>>> #PrivateMirror mirror1.mynetwork.com <http://mirror1.mynetwork.com/>
>>>> 
>>>> Try commenting that line out.
>>>> 
>>>>> *# freshclam*
>>>>> 
>>>>> *ClamAV update process started at Sat Jan 13 13:07:47 2018Reading CVD
>>>>> header (main.cld): OK (IMS)Reading CVD header (main.cvd): OK
>>>>> (IMS)main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60,
>>>>> builder: sigmgr)Reading CVD header (daily.cld): OKDownloading
>>>>> daily.cld [100%]daily.cld updated (version: 24217, sigs: 1823727,
>>>>> f-level: 63, builder: neo)Reading CVD header (bytecode.cld):
>>>>> OKbytecode.cvd is up to date (version: 319, sigs: 75, f-level: 63,
>>>>> builder: neo)[LibClamAV] Detected duplicate databases
>>>>> /var/lib/clamav/daily.cvd and /var/lib/clamav/daily.cld. The
>>>>> /var/lib/clamav/daily.cvd database is older and will not be loaded,
>>>>> you should manually remove it from the database directory.Database
>>>>> updated (6390051 signatures) from mirror.recette.local (IP:
>>>>> 192.168.1.15)*If I remove cld files from my private mirror :
>>>>> 
>>>>> *# freshclamClamAV update process started at Sat Jan 13 13:10:18
>>>>> 2018Downloading main.cld [100%]main.cld updated (version: 58, sigs:
>>>>> 4566249, f-level: 60, builder: sigmgr)Downloading daily.cld
>>>>> [100%]daily.cld updated (version: 24217, sigs: 1823727, f-level: 63,
>>>>> builder: neo)WARNING: getfile: bytecode.cld not found on
>>>>> mirror.recette.local (IP: 192.168.1.15)WARNING: Can't download
>>>>> bytecode.cld from mirror.recette.localDownloading bytecode.cvd
>>>>> [100%]bytecode.cvd updated (version: 319, sigs: 75, f-level: 63,
>>>>> builder: neo)Database updated (6390051 signatures) from
>>>>> mirror.recette.local (IP: 192.168.1.15)*
>>>>> 
>>>>> I just want to run freshclam without any error.
>>>>> 
>>>>> * Any idea ? *
>>>>> 
>>>>> On 11 January 2018 at 10:37, Al Varnell <alvarnell at mac.com <mailto:alvarnell at mac.com>> wrote:
>>>>> 
>>>>>> On Thu, Jan 11, 2018 at 01:15 AM, Hugo Deprez wrote:
>>>>>>> Hello,
>>>>>>> 
>>>>>>> thank you for the answer.
>>>>>>> I don't think my freshclam is trying to download the cdiff file
>>>> because I
>>>>>>> use : ScriptedUpdates no
>>>>>>> 
>>>>>>> Is this parameter you are talking about incremental updates ?
>>>>>> 
>>>>>> Yes, that's what I was referring to.
>>>>>> 
>>>>>>> I know this is not bandwith efficient, but I had to many issues with
>>>>>>> freshclam and my local repository made by clamdownloader.pl
>>>>>> 
>>>>>> So you are using an option 3 private local mirror as described in <
>>>>>> https://www.clamav.net/documents/private-local-mirrors <https://www.clamav.net/documents/private-local-mirrors>>. Make sure you
>>>>>> also change freshclam.conf "DatabaseMirror machine1.mylan" where
>>>>>> machine1.mylan is the name of your mirror server.
>>>>>> 
>>>>>> Also make sure you are using the latest version of the
>>>> clamavdownloader.pl
>>>>>> <https://github.com/akissa/clamav-faq/blob/master/ <https://github.com/akissa/clamav-faq/blob/master/>
>>>>>> mirrors/clamdownloader.pl>
>>>>>> 
>>>>>> The existing clamdownloader.pl script does not have any error
>>>> correction
>>>>>> it simply bails out if a downloaded file is not valid and is unable to
>>>>>> retry different mirrors if one fails. That is the most likely reason
>>>> for
>>>>>> those 404 errors.
>>>>>> 
>>>>>> I have no experience with private local mirrors, but you might have
>>>> better
>>>>>> luck with the clamavmirror script <https://pypi.python.org/pypi/ <https://pypi.python.org/pypi/>
>>>>>> clamavmirror/0.0.3>.
>>>>>> 
>>>>>> -Al-
>>>>>> 
>>>>>>> Best regards,
>>>>>>> 
>>>>>>> On 10 January 2018 at 10:20, Al Varnell <alvarnell at mac.com <mailto:alvarnell at mac.com> <mailto:
>>>>>> alvarnell at mac.com <mailto:alvarnell at mac.com>>> wrote:
>>>>>>> 
>>>>>>>> The first time freshclam pulls down a daily.cdiff file, your
>>>> daily.cvd
>>>>>>>> file will be decompressed to daily.cld and the .cdiff file added to
>>>> it.
>>>>>>>> From that point on you should only have the daily.cld file. One
>>>>>> exception
>>>>>>>> is that if for some reason freshclam is unable to find needed .cdiff
>>>>>> files
>>>>>>>> on a mirror, it will download a new .cvd file which will start the
>>>>>> process
>>>>>>>> again.
>>>>>>>> 
>>>>>>>> It is possible to reconfigure freshclam to download .cvd files by
>>>>>>>> disabling incremental updates, but that would be very inefficient
>>>> use of
>>>>>>>> bandwidth and mirror server time, so unless you have a overriding
>>>> need
>>>>>> for
>>>>>>>> that, I don't recommend it.
>>>>>>>> 
>>>>>>>> -Al-
>>>>>>>> 
>>>>>>>> On Wed, Jan 10, 2018 at 01:12 AM, Hugo Deprez wrote:
>>>>>>>>> Hello,
>>>>>>>>> 
>>>>>>>>> I have a question about daily.cvd and daily.cld files. If I
>>>> understood
>>>>>>>>> correctly, those two files are almost the same : one is compressed,
>>>> the
>>>>>>>>> other is not.
>>>>>>>>> Still I have an issue in my setup :
>>>>>>>>> 
>>>>>>>>> If I put both filtes in apache2 server (which act as PrivateMirror)
>>>> :
>>>>>>>>> 
>>>>>>>>> -rw-r--r-- 1 www-data www-data 117892267 07.06.2017 23:38 main.cvd
>>>>>>>>> -rw-r--r-- 1 www-data www-data    153228 07.12.2017 03:17
>>>> bytecode.cvd
>>>>>>>>> lrwxrwxrwx 1 root     root             8 05.01.2018 10:14 main.cld
>>>> ->
>>>>>>>>> main.cvd
>>>>>>>>> lrwxrwxrwx 1 root     root             9 05.01.2018 10:14 daily.cld
>>>> ->
>>>>>>>>> daily.cvd
>>>>>>>>> lrwxrwxrwx 1 root     root            12 05.01.2018 10:14
>>>> bytecode.cld
>>>>>> ->
>>>>>>>>> bytecode.cvd
>>>>>>>>> -rw-r--r-- 1 www-data www-data  43804052 10.01.2018 06:17 daily.cvd
>>>>>>>>> 
>>>>>>>>> On the client side I have this Warning :
>>>>>>>>> 
>>>>>>>>> LibClamAV Warning: Detected duplicate databases
>>>>>> /var/lib/clamav/daily.cvd
>>>>>>>>> and /var/lib/clamav/daily.cld, please manually remove one of them
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> But If I remove the *.cld files on my PrivateMirror I got multiples
>>>> 404
>>>>>>>>> errors from the Freshclam clients.
>>>>>>>>> 
>>>>>>>>> Is there a way to configure Freshclam in order to grabe only the cvd
>>>>>>>> files
>>>>>>>>> ?
>>>>>>>>> 
>>>>>>>>> Best regards,
>_______________________________________________
>clamav-users mailing list
>clamav-users at lists.clamav.net
>http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
>Help us build a comprehensive ClamAV guide:
>https://github.com/vrtadmin/clamav-faq
>
>http://www.clamav.net/contact.html#ml


More information about the clamav-users mailing list