Regarding slow scan times today (and slow scan times in general), it appears that the signatures we generate based on PhishTank’s feed for phishing URLs are resulting in very slow load and scan times.

Today’s daily update saw 7448 new Phishtank signatures (much higher than usual) coinciding with the immediate performance drop for load time and scan time. One user reported that the load time today on some of his slower machines was slow enough to exceed the timeout for service startup (https://bugzilla.clamav.net/show_bug.cgi?id=12317).

In limited testing on my own machine I saw the following change after dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:

Database load time on my laptop went from 75.43203997612 seconds down to 14.859203100204468 seconds
Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.

After some discussion between the teams that work on ClamAV and ClamAV signature content and deployment, we’ve agreed to drop PhishTank signatures from the database until we can determine a way to craft Phishtank signatures without incurring such a significant performance hit.

The daily update tomorrow will have the change.

-Micah

Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.

From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net>
Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
Date: Friday, April 5, 2019 at 1:08 PM
To: Mark Allan <markjallan@gmail.com>, ClamAV users ML <clamav-users@lists.clamav.net>
Cc: "Micah Snyder (micasnyd)" <micasnyd@cisco.com>
Subject: Re: [clamav-users] Scan very slow

Hi Mark,

Sorry about the delay in responding. I hadn’t looked at my clamav-users filter this morning. Just investigating now. Will respond when I know more.

-Micah

From: Mark Allan <markjallan@gmail.com>
Date: Friday, April 5, 2019 at 9:12 AM
To: ClamAV users ML <clamav-users@lists.clamav.net>, "Micah Snyder (micasnyd)" <micasnyd@cisco.com>
Subject: Re: [clamav-users] Scan very slow

Also CC'ing Micah directly as the mailing list would appear to be offline (at least lists.clamav.net isn't responding to http requests anyway)

It looks like scan times have gone through the roof. As Oya said, they're still considerably higher than they were a couple of months ago, but today's scan time is insane.

Yesterday's scan using

0.101.2:58:25409:1554370140:1:63:48554:328

took 7m 3s

On the same hardware, scanning the same read-only disk image, with today's scan using

0.101.2:58:25410:1554452941:1:63:48557:328

the scan time has jumped to 26m 15s

This is the longest it has ever taken to scan this volume (cf my previous email of 25th March)

Is there anything that can be excluded?

Best regards

Mark

On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <clamav-users@lists.clamav.net> wrote:

Thanks Oya for the update. We will continue to investigate the signature performance issue.

Regards,
Micah

On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <clamav-users-bounces@lists.clamav.net on behalf of oyamada@promark-inc.com> wrote:

Hi Micah

It seems that the scanning slow down issue of this time has been solved
at some level with CVD Update of the other day.
However, there is still big discrepancy in between the current condition and
the last condition in one month ago.

Date Files Scan time
2019/02/15 2550338 08:53:57
2019/03/15 2612792 19:22:54
2019/03/26 2634489 18:13:56
2019/03/27 2637201 18:10:05

We know the improvement of this time is due to the details of CVD, because
we did not make any change on the user's system.
We are going to try some tuning for scanning.

We like to know if you still have some room to make further improvement
for this slow down issue.
Thank you for your help, in advance.

Best regards,
Oya

On Mon, 25 Mar 2019 15:45:02 +0000
"Micah Snyder \(micasnyd\) via clamav-users" <clamav-users@lists.clamav.net> wrote:

> Hi Mark, all:
>
> I’m disappointed to hear that it is still slow for you.
>
> We found that the target-type of signatures used for PhishTank.Phishing signatures were causing a significant slowdown. We have dropped them as of this past Saturday ( https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates have been re-adding them with more specific scan target types. We’re now investigating some other optimizations we can make for the next major ClamAV release to improve scan times but at present we don’t have any other leads for signatures that may be slowing down scans.
>
> Regards,
> Micah
>
>
> From: clamav-users <clamav-users-bounces@lists.clamav.net> on behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net>
> Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
> Date: Monday, March 25, 2019 at 9:37 AM
> To: ClamAV users ML <clamav-users@lists.clamav.net>
> Cc: Mark Allan <markjallan@gmail.com>
> Subject: Re: [clamav-users] Scan very slow
>
> Cheers Steve,
>
> In the interest of completeness, here's the scan from today (TXT from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked improvement in scan time, although at 6m 7s it's still almost twice what it used to be.
>
> Mark
>
> On Mon, 25 Mar 2019 at 12:56, Steve Basford <steveb_clamav@sanesecurity.com<mailto:steveb_clamav@sanesecurity.com>> wrote:
> On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
> > Hi all,
> >
> te.
> >
> > Hopefully this helps someone to narrow things down a bit.
> >
> > Mark
> >
>
> 18/3/19 10m 49s TXT from DNS:
> 0.101.1:58:25392:1552904941:1:63:48507:328 ***
>
> Here's the changes for the above update:
>
> https://lists.gt.net/clamav/virusdb/75154
>
> You can also check sigs quickly per update:
>
> https://lists.gt.net/clamav/virusdb/
>
>
>
> --
> Cheers,
>
> Steve
> Twitter: @sanesecurity
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml