Maarten,Thanks for reporting that. There is an ordering difference of the content in the latest GDB file which is affecting the load time, and we will be fixing that in the next safebrowsing CVD version.Dave R.On Wed, Mar 6, 2019 at 10:42 AM Maarten Broekman via clamav-users <clamav-users@lists.clamav.net> wrote:I'm not sure if the safebrowsing.cld is included in the daily cdiff, but the current safebrowsing.cld takes between 50 and 70 seconds to load into clamscan, where a copy from February loads in <5 seconds.safebrowsing data:Old (fast): ClamAV-VDB:13 Feb 2019 13-16 -0500:48472:3041760:63:X:X:google:1550081775New (slow): ClamAV-VDB:05 Mar 2019 19-20 -0500:48473:3229612:63:X:X:google:1551831615Anyone know what might have changed in there to so drastically increased the load time?This happened after freshclam ran last night.# /opt/clamav/clamav/bin/clamscan -d ~/safebrowsing.cld samples/clam_test.htmlsamples/clam_test.html: OK----------- SCAN SUMMARY -----------Known viruses: 3041760Engine version: 0.100.2Scanned directories: 0Scanned files: 1Infected files: 0Data scanned: 0.00 MBData read: 0.00 MB (ratio 0.00:1)Time: 2.423 sec (0 m 2 s)# /opt/clamav/clamav/bin/clamscan -d /opt/clamav/var/lib/clamav/safebrowsing.cld samples/clam_test.htmlsamples/clam_test.html: OK----------- SCAN SUMMARY -----------Known viruses: 3229612Engine version: 0.100.2Scanned directories: 0Scanned files: 1Infected files: 0Data scanned: 0.00 MBData read: 0.00 MB (ratio 0.00:1)Time: 64.429 sec (1 m 4 s)On Wed, Mar 6, 2019 at 10:17 AM Micah Snyder (micasnyd) via clamav-users <clamav-users@lists.clamav.net> wrote:I confirmed with our signature management team that the extended time processing daily-25380 is because this change is significantly larger than a standard update.
This update drops 768053 hash-based signatures for malware that is detected by other more efficient logical signatures. The net result will be a leaner database that should load a little faster and take up less memory.
The validation stage when creating the daily had estimated less than 26 minutes for the cdiff to apply. You may be correct that it's much faster on x86 than on Sparc. 3h15m is definitely worse than expected, and I apologize for the inconvenience.
Regards,
Micah
Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.
On 3/6/19, 9:31 AM, "Pierre Dehaen" <dehaenp@drever.be> wrote:
Yes Micah, it finished while I was checking the computer because of the messages received
on the mailing list.
$ tail -50 /var/log/freshclam.log
...
--------------------------------------
ClamAV update process started at Wed Mar 6 11:37:46 2019
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.100.0 Recommended version: 0.101.1
DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
securiteinfo.hdb is up to date (version: custom database)
securiteinfo.ign2 is up to date (version: custom database)
Downloading javascript.ndb [*]
javascript.ndb updated (version: custom database, sigs: 45008)
securiteinfohtml.hdb is up to date (version: custom database)
securiteinfoascii.hdb is up to date (version: custom database)
securiteinfopdf.hdb is up to date (version: custom database)
Downloading spam_marketing.ndb [*]
spam_marketing.ndb updated (version: custom database, sigs: 24199)
main.cld is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Downloading daily-25380.cdiff [100%]
daily.cld updated (version: 25380, sigs: 1503528, f-level: 63, builder: raynman)
bytecode.cld is up to date (version: 328, sigs: 94, f-level: 63, builder: neo)
Database updated (6139078 signatures) from db.be.clamav.net (IP: 104.16.219.84)
Clamd successfully notified about the update.
$ ls -l /var/log/freshclam.log
-rw-r--r-- 1 clamav clamav 701634 Mar 6 14:51 /var/log/freshclam.log
It ran from 11:37 to 14:51. It might run faster on x86 computers though.
Pierre
On 6 Mar 2019 at 14:20, Micah Snyder (micasnyd) via clamav-users wrote:
Pierre,
So you're saying it actually did finish after 3 hours, 15 minutes on its own? That is good news
for all of the automated systems, even if this is a potentially terrible bug.
I'm still investigating the cause, and asking our signature management team if they have any
additional details.
Micah
Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.
On 3/6/19, 9:06 AM, "clamav-users on behalf of Pierre Dehaen" <clamav-users-
bounces@lists.clamav.net on behalf of dehaenp@drever.be> wrote:
Here too: it took about 3 hours and 15 minutes to calm down (SPARC, Solaris 11,
v0.100.0)... without noticiable error in freshclam.log.
On 6 Mar 2019 at 6:27, J.R. via clamav-users wrote:
> When crontab execs freshclam
> CPU server goes to 100%
> Hanged finishing Downloading daily-25380.cdiff [100%]
Just checked my server and it happened to me too! A little after 5am
central time. :(
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
--