[clamav-users] Problem with freshclam updating daily-25380.cdiff

David Raynor draynor at sourcefire.com
Wed Mar 6 16:57:39 EST 2019


>From my local validation with ClamAV, I can confidently say that
safebrowsing CVD 48474 will load much faster than safebrowsing CVD 48473.
It's all in the ordering. We'll be keeping track of that going forward.

Safebrowsing 48474 is available from the mirror network now.

Dave R.

On Wed, Mar 6, 2019 at 12:19 PM David Raynor <draynor at sourcefire.com> wrote:

> Maarten,
>
> Thanks for reporting that. There is an ordering difference of the content
> in the latest GDB file which is affecting the load time, and we will be
> fixing that in the next safebrowsing CVD version.
>
> Dave R.
>
> On Wed, Mar 6, 2019 at 10:42 AM Maarten Broekman via clamav-users <
> clamav-users at lists.clamav.net> wrote:
>
>> I'm not sure if the safebrowsing.cld is included in the daily cdiff, but
>> the current safebrowsing.cld takes between 50 and 70 seconds to *load* into
>> clamscan, where a copy from February loads in <5 seconds.
>>
>> safebrowsing data:
>> Old (fast):  ClamAV-VDB:13 Feb 2019 13-16
>> -0500:48472:3041760:63:X:X:google:1550081775
>>
>>
>> New (slow): ClamAV-VDB:05 Mar 2019 19-20
>> -0500:48473:3229612:63:X:X:google:1551831615
>>
>>
>>
>> Anyone know what might have changed in there to so drastically increased
>> the load time?
>>
>> This happened after freshclam ran last night.
>>
>> # /opt/clamav/clamav/bin/clamscan -d ~/safebrowsing.cld
>> samples/clam_test.html
>> samples/clam_test.html: OK
>>
>> ----------- SCAN SUMMARY -----------
>> Known viruses: 3041760
>> Engine version: 0.100.2
>> Scanned directories: 0
>> Scanned files: 1
>> Infected files: 0
>> Data scanned: 0.00 MB
>> Data read: 0.00 MB (ratio 0.00:1)
>> Time: 2.423 sec (0 m 2 s)
>>
>> # /opt/clamav/clamav/bin/clamscan -d
>> /opt/clamav/var/lib/clamav/safebrowsing.cld samples/clam_test.html
>> samples/clam_test.html: OK
>>
>> ----------- SCAN SUMMARY -----------
>> Known viruses: 3229612
>> Engine version: 0.100.2
>> Scanned directories: 0
>> Scanned files: 1
>> Infected files: 0
>> Data scanned: 0.00 MB
>> Data read: 0.00 MB (ratio 0.00:1)
>> Time: 64.429 sec (1 m 4 s)
>>
>>
>> On Wed, Mar 6, 2019 at 10:17 AM Micah Snyder (micasnyd) via clamav-users <
>> clamav-users at lists.clamav.net> wrote:
>>
>>> I confirmed with our signature management team that the extended time
>>> processing daily-25380 is because this change is significantly larger than
>>> a standard update.
>>> This update drops 768053 hash-based signatures for malware that is
>>> detected by other more efficient logical signatures.  The net result will
>>> be a leaner database that should load a little faster and take up less
>>> memory.
>>>
>>> The validation stage when creating the daily had estimated less than 26
>>> minutes for the cdiff to apply.  You may be correct that it's much faster
>>> on x86 than on Sparc.  3h15m is definitely worse than expected, and I
>>> apologize for the inconvenience.
>>>
>>> Regards,
>>> Micah
>>>
>>> Micah Snyder
>>> ClamAV Development
>>> Talos
>>> Cisco Systems, Inc.
>>>
>>>
>>> On 3/6/19, 9:31 AM, "Pierre Dehaen" <dehaenp at drever.be> wrote:
>>>
>>>     Yes Micah, it finished while I was checking the computer because of
>>> the messages received
>>>     on the mailing list.
>>>
>>>     $ tail -50 /var/log/freshclam.log
>>>     ...
>>>     --------------------------------------
>>>     ClamAV update process started at Wed Mar  6 11:37:46 2019
>>>     WARNING: Your ClamAV installation is OUTDATED!
>>>     WARNING: Local version: 0.100.0 Recommended version: 0.101.1
>>>     DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
>>>     securiteinfo.hdb is up to date (version: custom database)
>>>     securiteinfo.ign2 is up to date (version: custom database)
>>>     Downloading javascript.ndb [*]
>>>     javascript.ndb updated (version: custom database, sigs: 45008)
>>>     securiteinfohtml.hdb is up to date (version: custom database)
>>>     securiteinfoascii.hdb is up to date (version: custom database)
>>>     securiteinfopdf.hdb is up to date (version: custom database)
>>>     Downloading spam_marketing.ndb [*]
>>>     spam_marketing.ndb updated (version: custom database, sigs: 24199)
>>>     main.cld is up to date (version: 58, sigs: 4566249, f-level: 60,
>>> builder: sigmgr)
>>>     Downloading daily-25380.cdiff [100%]
>>>     daily.cld updated (version: 25380, sigs: 1503528, f-level: 63,
>>> builder: raynman)
>>>     bytecode.cld is up to date (version: 328, sigs: 94, f-level: 63,
>>> builder: neo)
>>>     Database updated (6139078 signatures) from db.be.clamav.net (IP:
>>> 104.16.219.84)
>>>     Clamd successfully notified about the update.
>>>
>>>     $ ls -l /var/log/freshclam.log
>>>     -rw-r--r--   1 clamav   clamav    701634 Mar  6 14:51
>>> /var/log/freshclam.log
>>>
>>>     It ran from 11:37 to 14:51. It might run faster on x86 computers
>>> though.
>>>
>>>     Pierre
>>>
>>>     On 6 Mar 2019 at 14:20, Micah Snyder (micasnyd) via clamav-users
>>> wrote:
>>>
>>>     Pierre,
>>>
>>>     So you're saying it actually did finish after 3 hours, 15 minutes on
>>> its own?  That is good news
>>>     for all of the automated systems, even if this is a potentially
>>> terrible bug.
>>>
>>>     I'm still investigating the cause, and asking our signature
>>> management team if they have any
>>>     additional details.
>>>
>>>     Micah
>>>
>>>     Micah Snyder
>>>     ClamAV Development
>>>     Talos
>>>     Cisco Systems, Inc.
>>>
>>>
>>>
>>>     On 3/6/19, 9:06 AM, "clamav-users on behalf of Pierre Dehaen"
>>> <clamav-users-
>>>     bounces at lists.clamav.net on behalf of dehaenp at drever.be> wrote:
>>>
>>>         Here too: it took about 3 hours and 15 minutes to calm down
>>> (SPARC, Solaris 11,
>>>         v0.100.0)... without noticiable error in freshclam.log.
>>>
>>>         On 6 Mar 2019 at 6:27, J.R. via clamav-users wrote:
>>>
>>>         > When crontab execs freshclam
>>>         > CPU server goes to 100%
>>>         > Hanged finishing Downloading daily-25380.cdiff [100%]
>>>
>>>         Just checked my server and it happened to me too! A little after
>>> 5am
>>>         central time.  :(
>>>
>>>         _______________________________________________
>>>
>>>         clamav-users mailing list
>>>         clamav-users at lists.clamav.net
>>>         https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>>         Help us build a comprehensive ClamAV guide:
>>>         https://github.com/vrtadmin/clamav-faq
>>>
>>>         http://www.clamav.net/contact.html#ml
>>>
>>>
>>>         _______________________________________________
>>>
>>>         clamav-users mailing list
>>>         clamav-users at lists.clamav.net
>>>         https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>>         Help us build a comprehensive ClamAV guide:
>>>         https://github.com/vrtadmin/clamav-faq
>>>
>>>         http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>>     _______________________________________________
>>>
>>>     clamav-users mailing list
>>>     clamav-users at lists.clamav.net
>>>     https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>>     Help us build a comprehensive ClamAV guide:
>>>     https://github.com/vrtadmin/clamav-faq
>>>
>>>     http://www.clamav.net/contact.html#ml
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> clamav-users mailing list
>>> clamav-users at lists.clamav.net
>>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>>
>>>
>>> Help us build a comprehensive ClamAV guide:
>>> https://github.com/vrtadmin/clamav-faq
>>>
>>> http://www.clamav.net/contact.html#ml
>>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users at lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>
>
> --
> ---
> Dave Raynor
> Talos Security Intelligence and Research Group
> draynor at sourcefire.com
>


-- 
---
Dave Raynor
Talos Security Intelligence and Research Group
draynor at sourcefire.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20190306/77860e43/attachment.html>


More information about the clamav-users mailing list