[clamav-users] Scan very slow

Fri Apr 5 19:17:01 UTC 2019

Regarding slow scan times today (and slow scan times in general), it appears that the signatures we generate based on PhishTank’s feed for phishing URLs are resulting in very slow load and scan times.

Today’s daily update saw 7448 new Phishtank signatures (much higher than usual) coinciding with the immediate performance drop for load time and scan time.  One user reported that the load time today on some of his slower machines was slow enough to exceed the timeout for service startup (https://bugzilla.clamav.net/show_bug.cgi?id=12317).

In limited testing on my own machine I saw the following change after dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:

  *   Database load time on my laptop went from 75.43203997612 seconds down to 14.859203100204468 seconds
  *   Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.

After some discussion between the teams that work on ClamAV and ClamAV signature content and deployment, we’ve agreed to drop PhishTank signatures from the database until we can determine a way to craft Phishtank signatures without incurring such a significant performance hit.

The daily update tomorrow will have the change.

-Micah

Micah Snyder
ClamAV Development
Talos
Cisco Systems, Inc.

From: clamav-users <clamav-users-bounces at lists.clamav.net> on behalf of "Micah Snyder (micasnyd) via clamav-users" <clamav-users at lists.clamav.net>
Reply-To: ClamAV users ML <clamav-users at lists.clamav.net>
Date: Friday, April 5, 2019 at 1:08 PM
To: Mark Allan <markjallan at gmail.com>, ClamAV users ML <clamav-users at lists.clamav.net>
Cc: "Micah Snyder (micasnyd)" <micasnyd at cisco.com>
Subject: Re: [clamav-users] Scan very slow

Hi Mark,

Sorry about the delay in responding.  I hadn’t looked at my clamav-users filter this morning.  Just investigating now.  Will respond when I know more.

-Micah

From: Mark Allan <markjallan at gmail.com>
Date: Friday, April 5, 2019 at 9:12 AM
To: ClamAV users ML <clamav-users at lists.clamav.net>, "Micah Snyder (micasnyd)" <micasnyd at cisco.com>
Subject: Re: [clamav-users] Scan very slow

Also CC'ing Micah directly as the mailing list would appear to be offline (at least lists.clamav.net<http://lists.clamav.net> isn't responding to http requests anyway)

It looks like scan times have gone through the roof. As Oya said, they're still considerably higher than they were a couple of months ago, but today's scan time is insane.

Yesterday's scan using
0.101.2:58:25409:1554370140:1:63:48554:328
took 7m 3s

On the same hardware, scanning the same read-only disk image, with today's scan using
0.101.2:58:25410:1554452941:1:63:48557:328
the scan time has jumped to 26m 15s

This is the longest it has ever taken to scan this volume (cf my previous email of 25th March)

Is there anything that can be excluded?

Best regards
Mark

On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>> wrote:
Thanks Oya for the update.  We will continue to investigate the signature performance issue.

Regards,
Micah

On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <clamav-users-bounces at lists.clamav.net<mailto:clamav-users-bounces at lists.clamav.net> on behalf of oyamada at promark-inc.com<mailto:oyamada at promark-inc.com>> wrote:

    Hi Micah

    It seems that the  scanning slow down issue of this time has been solved
    at some level with CVD Update of the other day.
    However, there is still big discrepancy in between the current condition and
    the last condition in one month ago.

    Date                Files               Scan time
    2019/02/15  2550338         08:53:57
    2019/03/15  2612792         19:22:54
    2019/03/26  2634489         18:13:56
    2019/03/27  2637201         18:10:05

    We know the improvement of this time is due to the details of CVD, because
    we did not make any change on the user's system.
    We are going to try some tuning for scanning.

    We like to know if you still have some room to make further improvement
    for this slow down issue.
    Thank you for your help, in advance.

    Best regards,
    Oya

    On Mon, 25 Mar 2019 15:45:02 +0000
    "Micah Snyder \(micasnyd\) via clamav-users" <clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>> wrote:

    > Hi Mark, all:
    >
    > I’m disappointed to hear that it is still slow for you.
    >
    > We found that the target-type of signatures used for PhishTank.Phishing signatures were causing a significant slowdown.   We have dropped them as of this past Saturday ( https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates have been re-adding them with more specific scan target types.  We’re now investigating some other optimizations we can make for the next major ClamAV release to improve scan times but at present we don’t have any other leads for signatures that may be slowing down scans.
    >
    > Regards,
    > Micah
    >
    >
    > From: clamav-users <clamav-users-bounces at lists.clamav.net<mailto:clamav-users-bounces at lists.clamav.net>> on behalf of Mark Allan via clamav-users <clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>>
    > Reply-To: ClamAV users ML <clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>>
    > Date: Monday, March 25, 2019 at 9:37 AM
    > To: ClamAV users ML <clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>>
    > Cc: Mark Allan <markjallan at gmail.com<mailto:markjallan at gmail.com>>
    > Subject: Re: [clamav-users] Scan very slow
    >
    > Cheers Steve,
    >
    > In the interest of completeness, here's the scan from today (TXT from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked improvement in scan time, although at 6m 7s it's still almost twice what it used to be.
    >
    > Mark
    >
    > On Mon, 25 Mar 2019 at 12:56, Steve Basford <steveb_clamav at sanesecurity.com<mailto:steveb_clamav at sanesecurity.com><mailto:steveb_clamav at sanesecurity.com<mailto:steveb_clamav at sanesecurity.com>>> wrote:
    > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
    > > Hi all,
    > >
    > te.
    > >
    > > Hopefully this helps someone to narrow things down a bit.
    > >
    > > Mark
    > >
    >
    > 18/3/19         10m 49s         TXT from DNS:
    > 0.101.1:58:25392:1552904941:1:63:48507:328      ***
    >
    > Here's the changes for the above update:
    >
    > https://lists.gt.net/clamav/virusdb/75154
    >
    > You can also check sigs quickly per update:
    >
    > https://lists.gt.net/clamav/virusdb/
    >
    >
    >
    > --
    > Cheers,
    >
    > Steve
    > Twitter: @sanesecurity
    >
    > _______________________________________________
    >
    > clamav-users mailing list
    > clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net><mailto:clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>>
    > https://lists.clamav.net/mailman/listinfo/clamav-users
    >
    >
    > Help us build a comprehensive ClamAV guide:
    > https://github.com/vrtadmin/clamav-faq
    >
    > http://www.clamav.net/contact.html#ml

    _______________________________________________

    clamav-users mailing list
    clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>
    https://lists.clamav.net/mailman/listinfo/clamav-users

    Help us build a comprehensive ClamAV guide:
    https://github.com/vrtadmin/clamav-faq

    http://www.clamav.net/contact.html#ml

_______________________________________________

clamav-users mailing list
clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>
https://lists.clamav.net/mailman/listinfo/clamav-users

Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20190405/7122e30e/attachment.htm>