[clamav-users] Detected duplicate databases

Hugo Deprez hugo.deprez at gmail.com
Mon Jan 22 09:46:35 UTC 2018


Hello,

thank you.

I reported a bug : https://bugzilla.clamav.net/show_bug.cgi?id=12009

If needed I can provide more informations.

Best regards



On 18 January 2018 at 23:23, Tom Judge (tomjudge) <tomjudge at cisco.com>
wrote:

> It sounds like at some point the freshclam process being used to populate
> the private mirror failed to delete the CVD file after applying a CDIFF and
> generating the CLD.
>
> The client in private mirror mode will look for both a CVD and a CLD and
> download the file with the correct version number after doing a HEAD for
> the metadata of the file.
>
> I'd start by deleting all the data from the mirror and re-initializing it
> with freshclam.  After 4 hours or so run freshclam again on the master to
> make sure it downloads and apples the CDIFF correctly and generates the CLD
> and removes the CLD.
>
> It's possible there is/was a permissions issue not allowing freshclam to
> delete the original CVD.
>
>
> Tom
>
> > On Jan 18, 2018, at 3:45 PM, Al Varnell <alvarnell at mac.com> wrote:
> >
> > As I mentioned before, I don't use a private mirror, so I've dropped out
> of the conversation. Someone who does use option 2 needs to double-check
> your settings here.
> >
> > -Al-
> >
> > On Thu, Jan 18, 2018 at 10:07 AM, Hugo Deprez wrote:
> >> Hello,
> >>
> >> I don't think I'm the only one having this issue. It seems that I used
> the
> >> official configuration from http://www.clamav.net/doc/mirrors-faq.html
> <http://www.clamav.net/doc/mirrors-faq.html>
> >> Should I report the issue somewhere ?
> >>
> >> The only way I see to avoid this issue is to create a shell script and
> to
> >> push the updates my self on all servers, but this is not the proper way
> to
> >> do this...
> >>
> >> Hugo
> >>
> >> On 15 January 2018 at 11:44, Hugo Deprez <hugo.deprez at gmail.com
> <mailto:hugo.deprez at gmail.com>> wrote:
> >>
> >>> Hello,
> >>>
> >>> by using this freshclam.conf  :
> >>>
> >>> AllowSupplementaryGroups false
> >>> Bytecode true
> >>> Checks 24
> >>> CompressLocalDatabase no
> >>> ConnectTimeout 30
> >>> DNSDatabaseInfo current.cvd.clamav.net <http://current.cvd.clamav.net/
> >
> >>> DatabaseDirectory /var/lib/clamav
> >>> DatabaseMirror mirror.recette.local
> >>> DatabaseOwner clamav
> >>> Debug false
> >>> Foreground false
> >>> LogFacility LOG_LOCAL6
> >>> LogFileMaxSize 0
> >>> LogRotate true
> >>> LogSyslog true
> >>> LogTime yes
> >>> LogVerbose false
> >>> MaxAttempts 5
> >>> PidFile /var/run/clamav/freshclam.pid
> >>> #PrivateMirror mirror.recette.local
> >>> ReceiveTimeout 30
> >>> ScriptedUpdates no
> >>> TestDatabases yes
> >>> UpdateLogFile /var/log/clamav/freshclam.log
> >>>
> >>> I got the old issue :
> >>>
> >>> root at server:/var/lib/clamav# rm -f *
> >>> root at server:/var/lib/clamav# freshclam
> >>> ClamAV update process started at Mon Jan 15 11:38:48 2018
> >>> Downloading main.cvd [100%]
> >>> main.cvd updated (version: 58, sigs: 4566249, f-level: 60, builder:
> sigmgr)
> >>> Downloading daily.cvd [100%]
> >>> daily.cvd updated (version: 24223, sigs: 1825336, f-level: 63, builder:
> >>> neo)
> >>> Downloading bytecode.cvd [100%]
> >>> bytecode.cvd updated (version: 315, sigs: 75, f-level: 63, builder:
> >>> raynman)
> >>> Database updated (6391660 signatures) from mirror.recette.local (IP:
> >>> 192.168.1.15)
> >>> root at raadmin:/var/lib/clamav# freshclam
> >>> ClamAV update process started at Mon Jan 15 11:40:00 2018
> >>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60,
> builder:
> >>> sigmgr)
> >>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
> >>> builder: neo)
> >>> Downloading bytecode.cvd [100%]
> >>> WARNING: Mirror 192.168.8.15 is not synchronized.
> >>> Trying again in 5 secs...
> >>> ClamAV update process started at Mon Jan 15 11:40:05 2018
> >>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60,
> builder:
> >>> sigmgr)
> >>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
> >>> builder: neo)
> >>> WARNING: Can't download bytecode.cvd from mirror.recette.local
> >>> Trying again in 5 secs...
> >>> ClamAV update process started at Mon Jan 15 11:40:10 2018
> >>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60,
> builder:
> >>> sigmgr)
> >>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
> >>> builder: neo)
> >>> WARNING: Can't download bytecode.cvd from mirror.recette.local
> >>> Trying again in 5 secs...
> >>> ClamAV update process started at Mon Jan 15 11:40:15 2018
> >>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60,
> builder:
> >>> sigmgr)
> >>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
> >>> builder: neo)
> >>> WARNING: Can't download bytecode.cvd from mirror.recette.local
> >>> Trying again in 5 secs...
> >>> ClamAV update process started at Mon Jan 15 11:40:20 2018
> >>> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60,
> builder:
> >>> sigmgr)
> >>> daily.cvd is up to date (version: 24223, sigs: 1825336, f-level: 63,
> >>> builder: neo)
> >>> ERROR: Can't download bytecode.cvd from mirror.recette.local
> >>> Giving up on mirror.recette.local ...
> >>> Update failed. Your network may be down or none of the mirrors listed
> in
> >>> /etc/clamav/freshclam.conf is working. Check
> http://www.clamav.net/doc/ <http://www.clamav.net/doc/>
> >>> mirrors-faq.html for possible reasons.
> >>>
> >>>
> >>>
> >>> On 14 January 2018 at 10:36, Al Varnell <alvarnell at mac.com <mailto:
> alvarnell at mac.com>> wrote:
> >>>
> >>>>
> >>>> On Sat, Jan 13, 2018 at 04:12 AM, Hugo Deprez wrote:
> >>>>>
> >>>>> hello,
> >>>>>
> >>>>> sorry I wasn't clear :
> >>>>>
> >>>>> - I used solution number 3 or more than one year, but freshclam on
> >>>> clients
> >>>>> was randomly reporting error on downloading cvd files. I never found
> the
> >>>>> solution :
> >>>>>
> >>>>> *Tue Dec  1 17:54:34 2015 -> WARNING: Can't download daily.cvd from *
> >>>>>
> >>>>> *mirror*
> >>>>>
> >>>>> - So now I switch to solution number 2 from
> >>>>> https://www.clamav.net/documents/private-local-mirrors <
> https://www.clamav.net/documents/private-local-mirrors>
> >>>>>
> >>>>> My "/etc/clamav/freshclam.conf" is :
> >>>>>
> >>>>> *AllowSupplementaryGroups falseBytecode trueChecks
> >>>>> 24CompressLocalDatabase noConnectTimeout 30DNSDatabaseInfo
> >>>>> current.cvd.clamav.net
> >>>>> <http://current.cvd.clamav.net>DatabaseDirectory
> >>>>> /var/lib/clamavDatabaseMirror mirror.recette.localDatabaseOwner
> >>>>> clamavDebug falseForeground falseLogFacility LOG_LOCAL6LogFileMaxSize
> >>>>> 0LogRotate trueLogSyslog trueLogTime yesLogVerbose falseMaxAttempts
> >>>>> 5PidFile /var/run/clamav/freshclam.pidPrivateMirror
> >>>>> mirror.recette.localReceiveTimeout 30ScriptedUpdates noTestDatabases
> >>>>> yesUpdateLogFile /var/log/clamav/freshclam.log*
> >>>>>
> >>>>> But when I run freshclam, it still downloading the cld files :
> >>>>
> >>>> Because you changed PrivateMirror to mirror.recette.local. See
> >>>> freshclam.conf:
> >>>>
> >>>> # This option allows you to easily point freshclam to private mirrors.
> >>>> # If PrivateMirror is set, freshclam does not attempt to use DNS
> >>>> # to determine whether its databases are out-of-date, instead it will
> >>>> # use the If-Modified-Since request or directly check the headers of
> the
> >>>> # remote database files. For each database, freshclam first attempts
> >>>> # to download the CLD file. If that fails, it tries to download the
> >>>> # CVD file. This option overrides DatabaseMirror, DNSDatabaseInfo
> >>>> # and ScriptedUpdates. It can be used multiple times to provide
> >>>> # fall-back mirrors.
> >>>> # Default: disabled
> >>>> #PrivateMirror mirror1.mynetwork.com <http://mirror1.mynetwork.com/>
> >>>>
> >>>> Try commenting that line out.
> >>>>
> >>>>> *# freshclam*
> >>>>>
> >>>>> *ClamAV update process started at Sat Jan 13 13:07:47 2018Reading CVD
> >>>>> header (main.cld): OK (IMS)Reading CVD header (main.cvd): OK
> >>>>> (IMS)main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60,
> >>>>> builder: sigmgr)Reading CVD header (daily.cld): OKDownloading
> >>>>> daily.cld [100%]daily.cld updated (version: 24217, sigs: 1823727,
> >>>>> f-level: 63, builder: neo)Reading CVD header (bytecode.cld):
> >>>>> OKbytecode.cvd is up to date (version: 319, sigs: 75, f-level: 63,
> >>>>> builder: neo)[LibClamAV] Detected duplicate databases
> >>>>> /var/lib/clamav/daily.cvd and /var/lib/clamav/daily.cld. The
> >>>>> /var/lib/clamav/daily.cvd database is older and will not be loaded,
> >>>>> you should manually remove it from the database directory.Database
> >>>>> updated (6390051 signatures) from mirror.recette.local (IP:
> >>>>> 192.168.1.15)*If I remove cld files from my private mirror :
> >>>>>
> >>>>> *# freshclamClamAV update process started at Sat Jan 13 13:10:18
> >>>>> 2018Downloading main.cld [100%]main.cld updated (version: 58, sigs:
> >>>>> 4566249, f-level: 60, builder: sigmgr)Downloading daily.cld
> >>>>> [100%]daily.cld updated (version: 24217, sigs: 1823727, f-level: 63,
> >>>>> builder: neo)WARNING: getfile: bytecode.cld not found on
> >>>>> mirror.recette.local (IP: 192.168.1.15)WARNING: Can't download
> >>>>> bytecode.cld from mirror.recette.localDownloading bytecode.cvd
> >>>>> [100%]bytecode.cvd updated (version: 319, sigs: 75, f-level: 63,
> >>>>> builder: neo)Database updated (6390051 signatures) from
> >>>>> mirror.recette.local (IP: 192.168.1.15)*
> >>>>>
> >>>>> I just want to run freshclam without any error.
> >>>>>
> >>>>> * Any idea ? *
> >>>>>
> >>>>> On 11 January 2018 at 10:37, Al Varnell <alvarnell at mac.com <mailto:
> alvarnell at mac.com>> wrote:
> >>>>>
> >>>>>> On Thu, Jan 11, 2018 at 01:15 AM, Hugo Deprez wrote:
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> thank you for the answer.
> >>>>>>> I don't think my freshclam is trying to download the cdiff file
> >>>> because I
> >>>>>>> use : ScriptedUpdates no
> >>>>>>>
> >>>>>>> Is this parameter you are talking about incremental updates ?
> >>>>>>
> >>>>>> Yes, that's what I was referring to.
> >>>>>>
> >>>>>>> I know this is not bandwith efficient, but I had to many issues
> with
> >>>>>>> freshclam and my local repository made by clamdownloader.pl
> >>>>>>
> >>>>>> So you are using an option 3 private local mirror as described in <
> >>>>>> https://www.clamav.net/documents/private-local-mirrors <
> https://www.clamav.net/documents/private-local-mirrors>>. Make sure you
> >>>>>> also change freshclam.conf "DatabaseMirror machine1.mylan" where
> >>>>>> machine1.mylan is the name of your mirror server.
> >>>>>>
> >>>>>> Also make sure you are using the latest version of the
> >>>> clamavdownloader.pl
> >>>>>> <https://github.com/akissa/clamav-faq/blob/master/ <
> https://github.com/akissa/clamav-faq/blob/master/>
> >>>>>> mirrors/clamdownloader.pl>
> >>>>>>
> >>>>>> The existing clamdownloader.pl script does not have any error
> >>>> correction
> >>>>>> it simply bails out if a downloaded file is not valid and is unable
> to
> >>>>>> retry different mirrors if one fails. That is the most likely reason
> >>>> for
> >>>>>> those 404 errors.
> >>>>>>
> >>>>>> I have no experience with private local mirrors, but you might have
> >>>> better
> >>>>>> luck with the clamavmirror script <https://pypi.python.org/pypi/ <
> https://pypi.python.org/pypi/>
> >>>>>> clamavmirror/0.0.3>.
> >>>>>>
> >>>>>> -Al-
> >>>>>>
> >>>>>>> Best regards,
> >>>>>>>
> >>>>>>> On 10 January 2018 at 10:20, Al Varnell <alvarnell at mac.com
> <mailto:alvarnell at mac.com> <mailto:
> >>>>>> alvarnell at mac.com <mailto:alvarnell at mac.com>>> wrote:
> >>>>>>>
> >>>>>>>> The first time freshclam pulls down a daily.cdiff file, your
> >>>> daily.cvd
> >>>>>>>> file will be decompressed to daily.cld and the .cdiff file added
> to
> >>>> it.
> >>>>>>>> From that point on you should only have the daily.cld file. One
> >>>>>> exception
> >>>>>>>> is that if for some reason freshclam is unable to find needed
> .cdiff
> >>>>>> files
> >>>>>>>> on a mirror, it will download a new .cvd file which will start the
> >>>>>> process
> >>>>>>>> again.
> >>>>>>>>
> >>>>>>>> It is possible to reconfigure freshclam to download .cvd files by
> >>>>>>>> disabling incremental updates, but that would be very inefficient
> >>>> use of
> >>>>>>>> bandwidth and mirror server time, so unless you have a overriding
> >>>> need
> >>>>>> for
> >>>>>>>> that, I don't recommend it.
> >>>>>>>>
> >>>>>>>> -Al-
> >>>>>>>>
> >>>>>>>> On Wed, Jan 10, 2018 at 01:12 AM, Hugo Deprez wrote:
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> I have a question about daily.cvd and daily.cld files. If I
> >>>> understood
> >>>>>>>>> correctly, those two files are almost the same : one is
> compressed,
> >>>> the
> >>>>>>>>> other is not.
> >>>>>>>>> Still I have an issue in my setup :
> >>>>>>>>>
> >>>>>>>>> If I put both filtes in apache2 server (which act as
> PrivateMirror)
> >>>> :
> >>>>>>>>>
> >>>>>>>>> -rw-r--r-- 1 www-data www-data 117892267 07.06.2017 23:38
> main.cvd
> >>>>>>>>> -rw-r--r-- 1 www-data www-data    153228 07.12.2017 03:17
> >>>> bytecode.cvd
> >>>>>>>>> lrwxrwxrwx 1 root     root             8 05.01.2018 10:14
> main.cld
> >>>> ->
> >>>>>>>>> main.cvd
> >>>>>>>>> lrwxrwxrwx 1 root     root             9 05.01.2018 10:14
> daily.cld
> >>>> ->
> >>>>>>>>> daily.cvd
> >>>>>>>>> lrwxrwxrwx 1 root     root            12 05.01.2018 10:14
> >>>> bytecode.cld
> >>>>>> ->
> >>>>>>>>> bytecode.cvd
> >>>>>>>>> -rw-r--r-- 1 www-data www-data  43804052 10.01.2018 06:17
> daily.cvd
> >>>>>>>>>
> >>>>>>>>> On the client side I have this Warning :
> >>>>>>>>>
> >>>>>>>>> LibClamAV Warning: Detected duplicate databases
> >>>>>> /var/lib/clamav/daily.cvd
> >>>>>>>>> and /var/lib/clamav/daily.cld, please manually remove one of them
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> But If I remove the *.cld files on my PrivateMirror I got
> multiples
> >>>> 404
> >>>>>>>>> errors from the Freshclam clients.
> >>>>>>>>>
> >>>>>>>>> Is there a way to configure Freshclam in order to grabe only the
> cvd
> >>>>>>>> files
> >>>>>>>>> ?
> >>>>>>>>>
> >>>>>>>>> Best regards,
> > _______________________________________________
> > clamav-users mailing list
> > clamav-users at lists.clamav.net
> > http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
>
>
> _______________________________________________
> clamav-users mailing list
> clamav-users at lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>



More information about the clamav-users mailing list