[clamav-users] [External] Re: Scan very slow
Brent Clark
brentgclarklist at gmail.com
Tue Apr 9 11:02:28 UTC 2019
Cant those be adopted / managed by Sanesecurity?
For all you know, those are already in Sanesecurity.
Regards
Brent Clark
On 2019/04/09 12:25, Mark Allan via clamav-users wrote:
> The scan times are definitely better than they were - in fact, they're
> back to how they were before last week's inclusion of the Phishtank
> signatures. They're still almost double what they used to be though, and
> as far as I can see, there are still almost 4000 Phishtank signatures in
> the DB:
> $ sigtool --find Phishtank | wc -l
> 3968
>
> Can I request that those ones also be removed please?
>
> Best regards
> Mark
>
> On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micasnyd at cisco.com
> <mailto:micasnyd at cisco.com>> wrote:
>
> Tim,____
>
> __ __
>
> There are a couple of ways for users to drop specific categories of
> signatures at this time. Sadly, they wouldn’t have helped this last
> week. These include bytecode signatures, PUA (potentially unwanted
> applications) signatures, Email.Phishing and HTML.Phishing
> signatures, and the Safebrowsing database. ____
>
> __ __
>
> If we had named the Phishtank.Phishing sigs to
> HTML.Phishing.Phishtank or Email.Phishing.Phishtank then they could
> have been disabled with the clamscan option `--phishing-sigs=no`
> (clamd.conf: `PhishingSignatures no`).____
>
> __ __
>
> Maybe a better option would be for us to create a new optional
> database for phishing signatures. However, the names for the
> databases are hardcoded into freshclam, so it is non-trivial to add
> a new database and would require a few changes to ClamAV’s code. We
> have talked about making the databases easier to add/remove in the
> future so users can have more categories to enable/disable. In this
> light, it ties in well with existing plans.____
>
> __ __
>
> Of note the Phishtank sigs from Friday’s daily were removed
> yesterday and scan times should be back to normal. ____
>
> __ __
>
> Regards,____
>
> Micah____
>
> __ __
>
> *From: *Tim Hawkins <tim.hawkins at redflaggroup.com
> <mailto:tim.hawkins at redflaggroup.com>>
> *Date: *Friday, April 5, 2019 at 6:06 PM
> *To: *ClamAV users ML <clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net>>, Mark Allan
> <markjallan at gmail.com <mailto:markjallan at gmail.com>>
> *Cc: *"Micah Snyder (micasnyd)" <micasnyd at cisco.com
> <mailto:micasnyd at cisco.com>>
> *Subject: *Re: [External] Re: [clamav-users] Scan very slow____
>
> __ __
>
> Hi Micah____
>
>
> Does clamav partition the database so that signatures that are
> mainly associated with email scanning can be dropped out for folks
> only needing filesystems scans, none of our systems use email, and
> we dont make use of the mailer extension.
>
> Having to load all the email focused signatures could as you have
> observed impact performance. ____
>
> Sent from Nine <http://www.9folders.com/>____
>
> ------------------------------------------------------------------------
>
> *From:* "Micah Snyder (micasnyd) via clamav-users"
> <clamav-users at lists.clamav.net <mailto:clamav-users at lists.clamav.net>>
> *Sent:* Saturday, April 6, 2019 03:18
> *To:* ClamAV users ML; Mark Allan
> *Cc:* Micah Snyder (micasnyd)
> *Subject:* [External] Re: [clamav-users] Scan very slow____
>
> __ __
>
> Regarding slow scan times today (and slow scan times in general), it
> appears that the signatures we generate based on PhishTank’s feed
> for phishing URLs are resulting in very slow load and scan times.____
>
> ____
>
> Today’s daily update saw 7448 new Phishtank signatures (much higher
> than usual) coinciding with the immediate performance drop for load
> time and scan time. One user reported that the load time today on
> some of his slower machines was slow enough to exceed the timeout
> for service startup
> (https://bugzilla.clamav.net/show_bug.cgi?id=12317).____
>
> ____
>
> In limited testing on my own machine I saw the following change
> after dropping the Phishtank.Phishing signatures from daily.cvd’s
> daily.ldb file:____
>
> * Database load time on my laptop went from 75.43203997612 seconds
> down to 14.859203100204468 seconds ____
> * Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644
> sec.____
>
> ____
>
> After some discussion between the teams that work on ClamAV and
> ClamAV signature content and deployment, we’ve agreed to drop
> PhishTank signatures from the database until we can determine a way
> to craft Phishtank signatures without incurring such a significant
> performance hit. ____
>
> ____
>
> The daily update tomorrow will have the change.____
>
> ____
>
> -Micah____
>
> ____
>
>
> Micah Snyder
> ClamAV Development
> Talos
> Cisco Systems, Inc.____
>
> ____
>
> ____
>
> ____
>
> *From: *clamav-users <clamav-users-bounces at lists.clamav.net
> <mailto:clamav-users-bounces at lists.clamav.net>> on behalf of "Micah
> Snyder (micasnyd) via clamav-users" <clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net>>
> *Reply-To: *ClamAV users ML <clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net>>
> *Date: *Friday, April 5, 2019 at 1:08 PM
> *To: *Mark Allan <markjallan at gmail.com
> <mailto:markjallan at gmail.com>>, ClamAV users ML
> <clamav-users at lists.clamav.net <mailto:clamav-users at lists.clamav.net>>
> *Cc: *"Micah Snyder (micasnyd)" <micasnyd at cisco.com
> <mailto:micasnyd at cisco.com>>
> *Subject: *Re: [clamav-users] Scan very slow____
>
> ____
>
> Hi Mark,____
>
> ____
>
> Sorry about the delay in responding. I hadn’t looked at my
> clamav-users filter this morning. Just investigating now. Will
> respond when I know more. ____
>
> ____
>
> -Micah____
>
> ____
>
> *From: *Mark Allan <markjallan at gmail.com <mailto:markjallan at gmail.com>>
> *Date: *Friday, April 5, 2019 at 9:12 AM
> *To: *ClamAV users ML <clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net>>, "Micah Snyder (micasnyd)"
> <micasnyd at cisco.com <mailto:micasnyd at cisco.com>>
> *Subject: *Re: [clamav-users] Scan very slow____
>
> ____
>
> Also CC'ing Micah directly as the mailing list would appear to be
> offline (at least lists.clamav.net <http://lists.clamav.net> isn't
> responding to http requests anyway) ____
>
> ____
>
> It looks like scan times have gone through the roof. As Oya said,
> they're still considerably higher than they were a couple of months
> ago, but today's scan time is insane.____
>
> ____
>
> Yesterday's scan using____
>
> 0.101.2:58:25409:1554370140:1:63:48554:328____
>
> took 7m 3s____
>
> ____
>
> On the same hardware, scanning the same read-only disk image, with
> today's scan using____
>
> 0.101.2:58:25410:1554452941:1:63:48557:328____
>
> the scan time has jumped to 26m 15s____
>
> ____
>
> This is the longest it has ever taken to scan this volume (cf my
> previous email of 25th March)____
>
> ____
>
> Is there anything that can be excluded?____
>
> ____
>
> Best regards____
>
> Mark____
>
> ____
>
> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via
> clamav-users <clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net>> wrote:____
>
> Thanks Oya for the update. We will continue to investigate the
> signature performance issue.
>
> Regards,
> Micah
>
> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada"
> <clamav-users-bounces at lists.clamav.net
> <mailto:clamav-users-bounces at lists.clamav.net> on behalf of
> oyamada at promark-inc.com <mailto:oyamada at promark-inc.com>> wrote:
>
> Hi Micah
>
> It seems that the scanning slow down issue of this time
> has been solved
> at some level with CVD Update of the other day.
> However, there is still big discrepancy in between the
> current condition and
> the last condition in one month ago.
>
> Date Files Scan time
> 2019/02/15 2550338 08:53:57
> 2019/03/15 2612792 19:22:54
> 2019/03/26 2634489 18:13:56
> 2019/03/27 2637201 18:10:05
>
> We know the improvement of this time is due to the details
> of CVD, because
> we did not make any change on the user's system.
> We are going to try some tuning for scanning.
>
> We like to know if you still have some room to make further
> improvement
> for this slow down issue.
> Thank you for your help, in advance.
>
> Best regards,
> Oya
>
> On Mon, 25 Mar 2019 15:45:02 +0000
> "Micah Snyder \(micasnyd\) via clamav-users"
> <clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net>> wrote:
>
> > Hi Mark, all:
> >
> > I’m disappointed to hear that it is still slow for you.
> >
> > We found that the target-type of signatures used for
> PhishTank.Phishing signatures were causing a significant
> slowdown. We have dropped them as of this past Saturday (
> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two
> updates have been re-adding them with more specific scan target
> types. We’re now investigating some other optimizations we can
> make for the next major ClamAV release to improve scan times but
> at present we don’t have any other leads for signatures that may
> be slowing down scans.
> >
> > Regards,
> > Micah
> >
> >
> > From: clamav-users <clamav-users-bounces at lists.clamav.net
> <mailto:clamav-users-bounces at lists.clamav.net>> on behalf of
> Mark Allan via clamav-users <clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net>>
> > Reply-To: ClamAV users ML <clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net>>
> > Date: Monday, March 25, 2019 at 9:37 AM
> > To: ClamAV users ML <clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net>>
> > Cc: Mark Allan <markjallan at gmail.com
> <mailto:markjallan at gmail.com>>
> > Subject: Re: [clamav-users] Scan very slow
> >
> > Cheers Steve,
> >
> > In the interest of completeness, here's the scan from
> today (TXT from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328)
> showing a marked improvement in scan time, although at 6m 7s
> it's still almost twice what it used to be.
> >
> > Mark
> >
> > On Mon, 25 Mar 2019 at 12:56, Steve Basford
> <steveb_clamav at sanesecurity.com
> <mailto:steveb_clamav at sanesecurity.com><mailto:steveb_clamav at sanesecurity.com
> <mailto:steveb_clamav at sanesecurity.com>>> wrote:
> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
> > > Hi all,
> > >
> > te.
> > >
> > > Hopefully this helps someone to narrow things down a bit.
> > >
> > > Mark
> > >
> >
> > 18/3/19 10m 49s TXT from DNS:
> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
> >
> > Here's the changes for the above update:
> >
> > https://lists.gt.net/clamav/virusdb/75154
> >
> > You can also check sigs quickly per update:
> >
> > https://lists.gt.net/clamav/virusdb/
> >
> >
> >
> > --
> > Cheers,
> >
> > Steve
> > Twitter: @sanesecurity
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net><mailto:clamav-users at lists.clamav.net
> <mailto:clamav-users at lists.clamav.net>>
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net <mailto:clamav-users at lists.clamav.net>
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net <mailto:clamav-users at lists.clamav.net>
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml____
>
>
>
> *DISCLAIMER*
>
> ____
>
> The information contained in this email and any attachments are
> confidential. It is intended solely for the individual or entity to
> whom they are addressed. Access to this email by anyone else is
> unauthorized.____
>
> If you are not the intended recipient, any disclosure, copying,
> distribution or any action taken or omitted to be taken in reliance
> on it, is prohibited and may be unlawful. If you have received this
> communication in error, please notify us immediately by responding
> to this email and then delete it from your system.____
>
> The Red Flag Group is neither liable for the proper and complete
> transmission of the information contained in this communication nor
> for any delay in its receipt.____
>
> Any advice, recommendations or opinion contained within this email
> or its attachments are not to be construed as legal advice.____
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
More information about the clamav-users
mailing list