[clamav-users] [External] Re: Scan very slow
Maarten Broekman
maarten.broekman at gmail.com
Wed Apr 17 10:36:56 UTC 2019
Are the "Phish" REPHISH signatures still in the daily or were they removed
as well? Those were causing part of the issue.
--Maarten
On Wed, Apr 17, 2019 at 5:24 AM Al Varnell via clamav-users <
clamav-users at lists.clamav.net> wrote:
> An additional 3968 Phishtank.Phishing.PHISH_ID_??????? signatures were
> dropped by daily-25417 on 12 April, and I can't seem to locate any more.
>
> -Al-
>
> On Apr 17, 2019, at 02:01, Mark Allan via clamav-users <
> clamav-users at lists.clamav.net> wrote:
>
> Hi Micah,
>
> Sorry to pester you, but have you any update on when the remaining
> Phishtank signatures will be getting removed? It would be really great to
> get scan times properly back to normal.
>
> Best regards
> Mark
>
> On Tue, 9 Apr 2019 at 16:32, Micah Snyder (micasnyd) <micasnyd at cisco.com>
> wrote:
>
>> Mark,
>>
>>
>> Yes, the plan is still to remove the rest of the Phishtank signatures.
>> We wanted to get things back to relative normal and resolve the immediate
>> crisis. We’ll remove the rest of them soon.
>>
>>
>>
>> Best,
>>
>> Micah
>>
>>
>>
>> *From: *Mark Allan <markjallan at gmail.com>
>> *Date: *Tuesday, April 9, 2019 at 6:26 AM
>> *To: *"Micah Snyder (micasnyd)" <micasnyd at cisco.com>
>> *Cc: *ClamAV users ML <clamav-users at lists.clamav.net>
>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>
>>
>>
>> The scan times are definitely better than they were - in fact, they're
>> back to how they were before last week's inclusion of the Phishtank
>> signatures. They're still almost double what they used to be though, and as
>> far as I can see, there are still almost 4000 Phishtank signatures in the
>> DB:
>>
>> $ sigtool --find Phishtank | wc -l
>>
>> 3968
>>
>>
>>
>> Can I request that those ones also be removed please?
>>
>>
>>
>> Best regards
>>
>> Mark
>>
>>
>>
>> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd at cisco.com>
>> wrote:
>>
>> Tim,
>>
>>
>>
>> There are a couple of ways for users to drop specific categories of
>> signatures at this time. Sadly, they wouldn’t have helped this last week.
>> These include bytecode signatures, PUA (potentially unwanted applications)
>> signatures, Email.Phishing and HTML.Phishing signatures, and the
>> Safebrowsing database.
>>
>>
>>
>> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or
>> Email.Phishing.Phishtank then they could have been disabled with the
>> clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).
>>
>>
>>
>> Maybe a better option would be for us to create a new optional database
>> for phishing signatures. However, the names for the databases are hardcoded
>> into freshclam, so it is non-trivial to add a new database and would
>> require a few changes to ClamAV’s code. We have talked about making the
>> databases easier to add/remove in the future so users can have more
>> categories to enable/disable. In this light, it ties in well with existing
>> plans.
>>
>>
>>
>> Of note the Phishtank sigs from Friday’s daily were removed yesterday and
>> scan times should be back to normal.
>>
>>
>>
>> Regards,
>>
>> Micah
>>
>>
>>
>> *From: *Tim Hawkins <tim.hawkins at redflaggroup.com>
>> *Date: *Friday, April 5, 2019 at 6:06 PM
>> *To: *ClamAV users ML <clamav-users at lists.clamav.net>, Mark Allan <
>> markjallan at gmail.com>
>> *Cc: *"Micah Snyder (micasnyd)" <micasnyd at cisco.com>
>> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>>
>>
>>
>> Hi Micah
>>
>>
>> Does clamav partition the database so that signatures that are mainly
>> associated with email scanning can be dropped out for folks only needing
>> filesystems scans, none of our systems use email, and we dont make use of
>> the mailer extension.
>>
>> Having to load all the email focused signatures could as you have
>> observed impact performance.
>>
>> Sent from Nine <http://www.9folders.com/>
>> ------------------------------
>>
>> *From:* "Micah Snyder (micasnyd) via clamav-users" <
>> clamav-users at lists.clamav.net>
>> *Sent:* Saturday, April 6, 2019 03:18
>> *To:* ClamAV users ML; Mark Allan
>> *Cc:* Micah Snyder (micasnyd)
>> *Subject:* [External] Re: [clamav-users] Scan very slow
>>
>>
>>
>> Regarding slow scan times today (and slow scan times in general), it
>> appears that the signatures we generate based on PhishTank’s feed for
>> phishing URLs are resulting in very slow load and scan times.
>>
>>
>>
>> Today’s daily update saw 7448 new Phishtank signatures (much higher than
>> usual) coinciding with the immediate performance drop for load time and
>> scan time. One user reported that the load time today on some of his
>> slower machines was slow enough to exceed the timeout for service startup (
>> https://bugzilla.clamav.net/show_bug.cgi?id=12317).
>>
>>
>>
>> In limited testing on my own machine I saw the following change after
>> dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:
>>
>> - Database load time on my laptop went from 75.43203997612 seconds
>> down to 14.859203100204468 seconds
>> - Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.
>>
>>
>>
>> After some discussion between the teams that work on ClamAV and ClamAV
>> signature content and deployment, we’ve agreed to drop PhishTank signatures
>> from the database until we can determine a way to craft Phishtank
>> signatures without incurring such a significant performance hit.
>>
>>
>>
>> The daily update tomorrow will have the change.
>>
>>
>>
>> -Micah
>>
>>
>>
>>
>> Micah Snyder
>> ClamAV Development
>> Talos
>> Cisco Systems, Inc.
>>
>>
>>
>>
>>
>>
>>
>> *From: *clamav-users <clamav-users-bounces at lists.clamav.net> on behalf
>> of "Micah Snyder (micasnyd) via clamav-users" <
>> clamav-users at lists.clamav.net>
>> *Reply-To: *ClamAV users ML <clamav-users at lists.clamav.net>
>> *Date: *Friday, April 5, 2019 at 1:08 PM
>> *To: *Mark Allan <markjallan at gmail.com>, ClamAV users ML <
>> clamav-users at lists.clamav.net>
>> *Cc: *"Micah Snyder (micasnyd)" <micasnyd at cisco.com>
>> *Subject: *Re: [clamav-users] Scan very slow
>>
>>
>>
>> Hi Mark,
>>
>>
>>
>> Sorry about the delay in responding. I hadn’t looked at my clamav-users
>> filter this morning. Just investigating now. Will respond when I know
>> more.
>>
>>
>>
>> -Micah
>>
>>
>>
>> *From: *Mark Allan <markjallan at gmail.com>
>> *Date: *Friday, April 5, 2019 at 9:12 AM
>> *To: *ClamAV users ML <clamav-users at lists.clamav.net>, "Micah Snyder
>> (micasnyd)" <micasnyd at cisco.com>
>> *Subject: *Re: [clamav-users] Scan very slow
>>
>>
>>
>> Also CC'ing Micah directly as the mailing list would appear to be offline
>> (at least lists.clamav.net isn't responding to http requests anyway)
>>
>>
>>
>> It looks like scan times have gone through the roof. As Oya said, they're
>> still considerably higher than they were a couple of months ago, but
>> today's scan time is insane.
>>
>>
>>
>> Yesterday's scan using
>>
>> 0.101.2:58:25409:1554370140:1:63:48554:328
>>
>> took 7m 3s
>>
>>
>>
>> On the same hardware, scanning the same read-only disk image, with
>> today's scan using
>>
>> 0.101.2:58:25410:1554452941:1:63:48557:328
>>
>> the scan time has jumped to 26m 15s
>>
>>
>>
>> This is the longest it has ever taken to scan this volume (cf my previous
>> email of 25th March)
>>
>>
>>
>> Is there anything that can be excluded?
>>
>>
>>
>> Best regards
>>
>> Mark
>>
>>
>>
>> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <
>> clamav-users at lists.clamav.net> wrote:
>>
>> Thanks Oya for the update. We will continue to investigate the signature
>> performance issue.
>>
>> Regards,
>> Micah
>>
>> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <
>> clamav-users-bounces at lists.clamav.net on behalf of
>> oyamada at promark-inc.com> wrote:
>>
>> Hi Micah
>>
>> It seems that the scanning slow down issue of this time has been
>> solved
>> at some level with CVD Update of the other day.
>> However, there is still big discrepancy in between the current
>> condition and
>> the last condition in one month ago.
>>
>> Date Files Scan time
>> 2019/02/15 2550338 08:53:57
>> 2019/03/15 2612792 19:22:54
>> 2019/03/26 2634489 18:13:56
>> 2019/03/27 2637201 18:10:05
>>
>> We know the improvement of this time is due to the details of CVD,
>> because
>> we did not make any change on the user's system.
>> We are going to try some tuning for scanning.
>>
>> We like to know if you still have some room to make further
>> improvement
>> for this slow down issue.
>> Thank you for your help, in advance.
>>
>> Best regards,
>> Oya
>>
>> On Mon, 25 Mar 2019 15:45:02 +0000
>> "Micah Snyder \(micasnyd\) via clamav-users" <
>> clamav-users at lists.clamav.net> wrote:
>>
>> > Hi Mark, all:
>> >
>> > I’m disappointed to hear that it is still slow for you.
>> >
>> > We found that the target-type of signatures used for
>> PhishTank.Phishing signatures were causing a significant slowdown. We
>> have dropped them as of this past Saturday (
>> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates
>> have been re-adding them with more specific scan target types. We’re now
>> investigating some other optimizations we can make for the next major
>> ClamAV release to improve scan times but at present we don’t have any other
>> leads for signatures that may be slowing down scans.
>> >
>> > Regards,
>> > Micah
>> >
>> >
>> > From: clamav-users <clamav-users-bounces at lists.clamav.net> on
>> behalf of Mark Allan via clamav-users <clamav-users at lists.clamav.net>
>> > Reply-To: ClamAV users ML <clamav-users at lists.clamav.net>
>> > Date: Monday, March 25, 2019 at 9:37 AM
>> > To: ClamAV users ML <clamav-users at lists.clamav.net>
>> > Cc: Mark Allan <markjallan at gmail.com>
>> > Subject: Re: [clamav-users] Scan very slow
>> >
>> > Cheers Steve,
>> >
>> > In the interest of completeness, here's the scan from today (TXT
>> from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked
>> improvement in scan time, although at 6m 7s it's still almost twice what it
>> used to be.
>> >
>> > Mark
>> >
>> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <
>> steveb_clamav at sanesecurity.com<mailto:steveb_clamav at sanesecurity.com>>
>> wrote:
>> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
>> > > Hi all,
>> > >
>> > te.
>> > >
>> > > Hopefully this helps someone to narrow things down a bit.
>> > >
>> > > Mark
>> > >
>> >
>> > 18/3/19 10m 49s TXT from DNS:
>> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
>> >
>> > Here's the changes for the above update:
>> >
>> > https://lists.gt.net/clamav/virusdb/75154
>> >
>> > You can also check sigs quickly per update:
>> >
>> > https://lists.gt.net/clamav/virusdb/
>> >
>> >
>> >
>> > --
>> > Cheers,
>> >
>> > Steve
>> > Twitter: @sanesecurity
>> >
>> > _______________________________________________
>> >
>> > clamav-users mailing list
>> > clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>
>> > https://lists.clamav.net/mailman/listinfo/clamav-users
>> >
>> >
>> > Help us build a comprehensive ClamAV guide:
>> > https://github.com/vrtadmin/clamav-faq
>> >
>> > http://www.clamav.net/contact.html#ml
>>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users at lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users at lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>>
>>
>> *DISCLAIMER*
>>
>> The information contained in this email and any attachments are
>> confidential. It is intended solely for the individual or entity to whom
>> they are addressed. Access to this email by anyone else is unauthorized.
>>
>> If you are not the intended recipient, any disclosure, copying,
>> distribution or any action taken or omitted to be taken in reliance on it,
>> is prohibited and may be unlawful. If you have received this communication
>> in error, please notify us immediately by responding to this email and then
>> delete it from your system.
>>
>> The Red Flag Group is neither liable for the proper and complete
>> transmission of the information contained in this communication nor for any
>> delay in its receipt.
>>
>> Any advice, recommendations or opinion contained within this email or its
>> attachments are not to be construed as legal advice.
>>
>>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20190417/13f49fec/attachment.htm>
More information about the clamav-users
mailing list