Clearly the latest daily.cvd is performing better, but the remaining "Phishtank" sigs are not a majority of the slowness.

I unpacked the current (?) cvd (ClamAV-VDB:09 Apr 2019 03-53 -0400:25414:1548262:63:X:X:raynman:1554796413) and then ran a test scan with each part to see what the load times looked like:
daily.cdb ==== Time: 0.007 sec (0 m 0 s)
daily.cfg ==== Time: 0.004 sec (0 m 0 s)
daily.crb ==== Time: 0.006 sec (0 m 0 s)
daily.cvd ==== Time: 11.384 sec (0 m 11 s)
daily.fp ==== Time: 0.009 sec (0 m 0 s)
daily.ftm ==== Time: 0.005 sec (0 m 0 s)
daily.hdb ==== Time: 0.303 sec (0 m 0 s)
daily.hdu ==== Time: 0.006 sec (0 m 0 s)
daily.hsb ==== Time: 1.093 sec (0 m 1 s)
daily.hsu ==== Time: 0.005 sec (0 m 0 s)
daily.idb ==== Time: 0.006 sec (0 m 0 s)
daily.ldb ==== Time: 5.563 sec (0 m 5 s)
daily.ldu ==== Time: 0.005 sec (0 m 0 s)
daily.mdb ==== Time: 0.061 sec (0 m 0 s)
daily.mdu ==== Time: 0.007 sec (0 m 0 s)
daily.msb ==== Time: 0.005 sec (0 m 0 s)
daily.msu ==== Time: 0.005 sec (0 m 0 s)
daily.ndb ==== Time: 0.017 sec (0 m 0 s)
daily.ndu ==== Time: 0.005 sec (0 m 0 s)
daily.pdb ==== Time: 0.010 sec (0 m 0 s)
daily.sfp ==== Time: 0.006 sec (0 m 0 s)
daily.wdb ==== Time: 0.014 sec (0 m 0 s)

So, half the run time of a clamscan is from the daily.ldb. To break it down farther, I split the daily.ldb into "daily_<virus>.ldb" where <virus> is the first part of the dot-separated signature name.
daily_Andr.ldb ==== Time: 0.008 sec (0 m 0 s)
daily_Archive.ldb ==== Time: 0.009 sec (0 m 0 s)
daily_Asp.ldb ==== Time: 0.004 sec (0 m 0 s)
daily_Doc.ldb ==== Time: 0.116 sec (0 m 0 s)
daily_Eicar-Test-Signature.ldb ==== Time: 0.009 sec (0 m 0 s)
daily_Email.ldb ==== Time: 0.014 sec (0 m 0 s)
daily_Emf.ldb ==== Time: 0.007 sec (0 m 0 s)
daily_Heuristics.ldb ==== Time: 0.006 sec (0 m 0 s)
daily_Html.ldb ==== Time: 0.010 sec (0 m 0 s)
daily_Hwp.ldb ==== Time: 0.005 sec (0 m 0 s)
daily_Img.ldb ==== Time: 0.006 sec (0 m 0 s)
daily_Ios.ldb ==== Time: 0.006 sec (0 m 0 s)
daily_Java.ldb ==== Time: 0.005 sec (0 m 0 s)
daily_Js.ldb ==== Time: 0.007 sec (0 m 0 s)
daily_Legacy.ldb ==== Time: 0.006 sec (0 m 0 s)
daily_Lnk.ldb ==== Time: 0.005 sec (0 m 0 s)
daily_Mp4.ldb ==== Time: 0.005 sec (0 m 0 s)
daily_Multios.ldb ==== Time: 0.005 sec (0 m 0 s)
daily_Osx.ldb ==== Time: 0.008 sec (0 m 0 s)
daily_Pdf.ldb ==== Time: 0.007 sec (0 m 0 s)
daily_Phish.ldb ==== Time: 1.612 sec (0 m 1 s)
daily_Phishtank.ldb ==== Time: 0.146 sec (0 m 0 s)
daily_Php.ldb ==== Time: 0.006 sec (0 m 0 s)
daily_Ppt.ldb ==== Time: 0.007 sec (0 m 0 s)
daily_Py.ldb ==== Time: 0.006 sec (0 m 0 s)
daily_Rtf.ldb ==== Time: 0.006 sec (0 m 0 s)
daily_Svg.ldb ==== Time: 0.005 sec (0 m 0 s)
daily_Swf.ldb ==== Time: 0.007 sec (0 m 0 s)
daily_Ttf.ldb ==== Time: 0.005 sec (0 m 0 s)
daily_Txt.ldb ==== Time: 0.009 sec (0 m 0 s)
daily_Unix.ldb ==== Time: 0.008 sec (0 m 0 s)
daily_Vbs.ldb ==== Time: 0.009 sec (0 m 0 s)
daily_Win.ldb ==== Time: 3.391 sec (0 m 3 s)
daily_Xls.ldb ==== Time: 0.009 sec (0 m 0 s)
daily_Xml.ldb ==== Time: 0.007 sec (0 m 0 s)

"Phish.", not "Phishtank.", and "Win." are the longest run times. Looking at the number of signatures in each, the 'Phish.' signatures are taking a disproportionate amount of time to load compared to the other signatures:
     216 daily_Andr.ldb
       3 daily_Archive.ldb
       1 daily_Asp.ldb
    2096 daily_Doc.ldb
       1 daily_Eicar-Test-Signature.ldb
    1017 daily_Email.ldb
       2 daily_Emf.ldb
       5 daily_Heuristics.ldb
     250 daily_Html.ldb
       1 daily_Hwp.ldb
      15 daily_Img.ldb
       6 daily_Ios.ldb
      16 daily_Java.ldb
      69 daily_Js.ldb
      27 daily_Legacy.ldb
       9 daily_Lnk.ldb
       1 daily_Mp4.ldb
       9 daily_Multios.ldb
     175 daily_Osx.ldb
     132 daily_Pdf.ldb
    2515 daily_Phish.ldb
    3516 daily_Phishtank.ldb
      18 daily_Php.ldb
       5 daily_Ppt.ldb
       3 daily_Py.ldb
      28 daily_Rtf.ldb
       1 daily_Svg.ldb
     103 daily_Swf.ldb
       2 daily_Ttf.ldb
     140 daily_Txt.ldb
     222 daily_Unix.ldb
      21 daily_Vbs.ldb
   43928 daily_Win.ldb
     165 daily_Xls.ldb
       8 daily_Xml.ldb

From the look of it, "Phish." has those REPHISH signatures. Those signatures seem to be looking at any file (Target 0) and have subsignatures that are combined to match depending on which filetype they are 'looking' for (so, href for HTML files, %PDF, Subtype, and URI objects for PDFs, etc) as opposed to the remaining Phishtank sigs which seem to have a separate signature depending on the target type.

Breaking up daily_Win into it's constituent sub-parts doesn't reveal any particular culprit from just a simple scan timing though...
daily_Win.Adware.ldb ==== Time: 0.013 sec (0 m 0 s)
daily_Win.Coinminer.ldb ==== Time: 0.009 sec (0 m 0 s)
daily_Win.Downloader.ldb ==== Time: 0.035 sec (0 m 0 s)
daily_Win.Dropper.ldb ==== Time: 0.240 sec (0 m 0 s)
daily_Win.Exploit.ldb ==== Time: 0.016 sec (0 m 0 s)
daily_Win.Ircbot.ldb ==== Time: 0.006 sec (0 m 0 s)
daily_Win.Keylogger.ldb ==== Time: 0.010 sec (0 m 0 s)
daily_Win.ldb ==== Time: 3.418 sec (0 m 3 s)
daily_Win.Macro.ldb ==== Time: 0.009 sec (0 m 0 s)
daily_Win.Malware.ldb ==== Time: 0.731 sec (0 m 0 s)
daily_Win.Packed.ldb ==== Time: 0.131 sec (0 m 0 s)
daily_Win.Packer.ldb ==== Time: 0.008 sec (0 m 0 s)
daily_Win.Phishing.ldb ==== Time: 0.005 sec (0 m 0 s)
daily_Win.Proxy.ldb ==== Time: 0.005 sec (0 m 0 s)
daily_Win.Ransomware.ldb ==== Time: 0.019 sec (0 m 0 s)
daily_Win.Spyware.ldb ==== Time: 0.006 sec (0 m 0 s)
daily_Win.Tool.ldb ==== Time: 0.008 sec (0 m 0 s)
daily_Win.Trojan.ldb ==== Time: 0.582 sec (0 m 0 s)
daily_Win.Virus.ldb ==== Time: 0.059 sec (0 m 0 s)
daily_Win.Worm.ldb ==== Time: 0.030 sec (0 m 0 s)

     158 daily_Win.Adware.ldb
      14 daily_Win.Coinminer.ldb
     561 daily_Win.Downloader.ldb
    8084 daily_Win.Dropper.ldb
     216 daily_Win.Exploit.ldb
       9 daily_Win.Ircbot.ldb
     193 daily_Win.Keylogger.ldb
   43928 daily_Win.ldb
       1 daily_Win.Macro.ldb
   14820 daily_Win.Malware.ldb
    4209 daily_Win.Packed.ldb
      20 daily_Win.Packer.ldb
       2 daily_Win.Phishing.ldb
       4 daily_Win.Proxy.ldb
     500 daily_Win.Ransomware.ldb
      32 daily_Win.Spyware.ldb
     121 daily_Win.Tool.ldb
   12051 daily_Win.Trojan.ldb
    1967 daily_Win.Virus.ldb
     966 daily_Win.Worm.ldb

Malware and Trojan take the longest, but they also have a majority of the signatures.

On Tue, Apr 9, 2019 at 11:19 AM Steve Basford <steveb_clamav@sanesecurity.com> wrote:
On 2019-04-09 12:02, Brent Clark via clamav-users wrote:
> Cant those be adopted / managed by Sanesecurity?
>
> For all you know, those are already in Sanesecurity.

They are... and have been for quite some time:


"The following databases are distributed by Sanesecurity, but produced
by Porcupine Signatures"

phishtank.ndb.

Briefly...

Number of sigs in phishtank.ndb: 9,309

eg:

PhishTank.Phishing.6002281, matches:

https://www.phishtank.com/phish_detail.php?phish_id=6002281

So, there is going to be some possible cross over now that
Phish.Phishing.REPHISH_ID_20190404_67-6931549-0
type signatures names from PhishTank feed are in daily.ldb and
daily.ndb.

I'll check back on the thread later.

--
Cheers,

Steve
Twitter: @sanesecurity

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml