[clamav-users] Problem with freshclam updating daily-25380.cdiff

Maarten Broekman maarten.broekman at gmail.com
Wed Mar 6 10:41:29 EST 2019


I'm not sure if the safebrowsing.cld is included in the daily cdiff, but
the current safebrowsing.cld takes between 50 and 70 seconds to *load* into
clamscan, where a copy from February loads in <5 seconds.

safebrowsing data:
Old (fast):  ClamAV-VDB:13 Feb 2019 13-16
-0500:48472:3041760:63:X:X:google:1550081775


New (slow): ClamAV-VDB:05 Mar 2019 19-20
-0500:48473:3229612:63:X:X:google:1551831615



Anyone know what might have changed in there to so drastically increased
the load time?

This happened after freshclam ran last night.

# /opt/clamav/clamav/bin/clamscan -d ~/safebrowsing.cld
samples/clam_test.html
samples/clam_test.html: OK

----------- SCAN SUMMARY -----------
Known viruses: 3041760
Engine version: 0.100.2
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 2.423 sec (0 m 2 s)

# /opt/clamav/clamav/bin/clamscan -d
/opt/clamav/var/lib/clamav/safebrowsing.cld samples/clam_test.html
samples/clam_test.html: OK

----------- SCAN SUMMARY -----------
Known viruses: 3229612
Engine version: 0.100.2
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 64.429 sec (1 m 4 s)


On Wed, Mar 6, 2019 at 10:17 AM Micah Snyder (micasnyd) via clamav-users <
clamav-users at lists.clamav.net> wrote:

> I confirmed with our signature management team that the extended time
> processing daily-25380 is because this change is significantly larger than
> a standard update.
> This update drops 768053 hash-based signatures for malware that is
> detected by other more efficient logical signatures.  The net result will
> be a leaner database that should load a little faster and take up less
> memory.
>
> The validation stage when creating the daily had estimated less than 26
> minutes for the cdiff to apply.  You may be correct that it's much faster
> on x86 than on Sparc.  3h15m is definitely worse than expected, and I
> apologize for the inconvenience.
>
> Regards,
> Micah
>
> Micah Snyder
> ClamAV Development
> Talos
> Cisco Systems, Inc.
>
>
> On 3/6/19, 9:31 AM, "Pierre Dehaen" <dehaenp at drever.be> wrote:
>
>     Yes Micah, it finished while I was checking the computer because of
> the messages received
>     on the mailing list.
>
>     $ tail -50 /var/log/freshclam.log
>     ...
>     --------------------------------------
>     ClamAV update process started at Wed Mar  6 11:37:46 2019
>     WARNING: Your ClamAV installation is OUTDATED!
>     WARNING: Local version: 0.100.0 Recommended version: 0.101.1
>     DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
>     securiteinfo.hdb is up to date (version: custom database)
>     securiteinfo.ign2 is up to date (version: custom database)
>     Downloading javascript.ndb [*]
>     javascript.ndb updated (version: custom database, sigs: 45008)
>     securiteinfohtml.hdb is up to date (version: custom database)
>     securiteinfoascii.hdb is up to date (version: custom database)
>     securiteinfopdf.hdb is up to date (version: custom database)
>     Downloading spam_marketing.ndb [*]
>     spam_marketing.ndb updated (version: custom database, sigs: 24199)
>     main.cld is up to date (version: 58, sigs: 4566249, f-level: 60,
> builder: sigmgr)
>     Downloading daily-25380.cdiff [100%]
>     daily.cld updated (version: 25380, sigs: 1503528, f-level: 63,
> builder: raynman)
>     bytecode.cld is up to date (version: 328, sigs: 94, f-level: 63,
> builder: neo)
>     Database updated (6139078 signatures) from db.be.clamav.net (IP:
> 104.16.219.84)
>     Clamd successfully notified about the update.
>
>     $ ls -l /var/log/freshclam.log
>     -rw-r--r--   1 clamav   clamav    701634 Mar  6 14:51
> /var/log/freshclam.log
>
>     It ran from 11:37 to 14:51. It might run faster on x86 computers
> though.
>
>     Pierre
>
>     On 6 Mar 2019 at 14:20, Micah Snyder (micasnyd) via clamav-users wrote:
>
>     Pierre,
>
>     So you're saying it actually did finish after 3 hours, 15 minutes on
> its own?  That is good news
>     for all of the automated systems, even if this is a potentially
> terrible bug.
>
>     I'm still investigating the cause, and asking our signature management
> team if they have any
>     additional details.
>
>     Micah
>
>     Micah Snyder
>     ClamAV Development
>     Talos
>     Cisco Systems, Inc.
>
>
>
>     On 3/6/19, 9:06 AM, "clamav-users on behalf of Pierre Dehaen"
> <clamav-users-
>     bounces at lists.clamav.net on behalf of dehaenp at drever.be> wrote:
>
>         Here too: it took about 3 hours and 15 minutes to calm down
> (SPARC, Solaris 11,
>         v0.100.0)... without noticiable error in freshclam.log.
>
>         On 6 Mar 2019 at 6:27, J.R. via clamav-users wrote:
>
>         > When crontab execs freshclam
>         > CPU server goes to 100%
>         > Hanged finishing Downloading daily-25380.cdiff [100%]
>
>         Just checked my server and it happened to me too! A little after
> 5am
>         central time.  :(
>
>         _______________________________________________
>
>         clamav-users mailing list
>         clamav-users at lists.clamav.net
>         https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
>         Help us build a comprehensive ClamAV guide:
>         https://github.com/vrtadmin/clamav-faq
>
>         http://www.clamav.net/contact.html#ml
>
>
>         _______________________________________________
>
>         clamav-users mailing list
>         clamav-users at lists.clamav.net
>         https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
>         Help us build a comprehensive ClamAV guide:
>         https://github.com/vrtadmin/clamav-faq
>
>         http://www.clamav.net/contact.html#ml
>
>
>
>     _______________________________________________
>
>     clamav-users mailing list
>     clamav-users at lists.clamav.net
>     https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
>     Help us build a comprehensive ClamAV guide:
>     https://github.com/vrtadmin/clamav-faq
>
>     http://www.clamav.net/contact.html#ml
>
>
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20190306/d414e0aa/attachment.html>


More information about the clamav-users mailing list