[clamav-users] Problem with new safebrowsing file

David Raynor draynor at sourcefire.com
Wed Mar 6 17:45:51 EST 2019


That's strange, the 48474 I have should have the sorting changed and has
the improved loading time we're talking about.

$ sigtool --info safebrowsing.cvd
File: safebrowsing.cvd
Build time: 06 Mar 2019 13:24 -0500
Version: 48474
Signatures: 3232286
Functionality level: 63
Builder: google
MD5: 70c61f41e52b5a2134ff7e272f5a6df1

SHA256 (safebrowsing.gdb) =
7f6645b8d865de3992be1ad5de215afd848acee4c021eed4818fdb760f76b57e

Something must be different.

Dave R.

On Wed, Mar 6, 2019 at 5:39 PM Maarten Broekman via clamav-users <
clamav-users at lists.clamav.net> wrote:

> The new safebrowsing cvd (starting with version 48473) seems to be sorted
> in a way that increases the load time of that file by several orders of
> magnitude.
>
> I have a previous version from February where the entries in the gdb
> section are sorted like this:
>
> S2:F:0000917787cff7b0993917209809ff3d94bec7e1de7188b323d9b88e0273cb71
> S2:F:000149794d90dc5bce4f685deed6076d00c9209bd81cef4cbdf8a4e41f0a2153
> S2:F:00042c895c912fd567afa35450cfe5d321d0d68eb3833156925c4e27d2c29aa2
> S2:F:0006d4dcb0d939d725e676a9e68aaeb303e04478e6861d2a77469d1b6a0a0f7d
> S2:F:0007bf7c1808d12177f0ae90d336d60c5a7a3d89703806955b75c56f898dd919
> ...
> S2:P:00009177
> S2:P:00014979
> S2:P:00042c89
> S2:P:0006d4dc
> S2:P:0007bf7c
> ...
> S:F:00000860493997b798861956e06d3d3606f82384259b971bb922f94f886a4b55
> S:F:00000bddafae162a7a2f1249b3b38c8e4b6d3cb8bf0c30c26cc354ebcba16b37
> S:F:000046cad35fbecbcc8dd4ebb244bd08aa6dbf1078279115c82f8e21b2cf8478
> S:F:0000684200da7b11f38a6f4719bda4ec6c6ae8b2be1f7e12a16605b2d3a5d490
> S:F:000072f3f33e47a2f97b8711d240267462aa3f0a5f8130845b119a2ad3798292
> ...
> S:P:00000860
> S:P:00000bdd
> S:P:000046ca
> S:P:00006842
> S:P:000072f3
>
>
> That loads into clamd (and clamscan) in under 5 seconds for the 3041760
> entries in it.
>
> Version 48473 and 48474 are sorted like this:
>
> S2:P:00009177
> S2:F:0000917787cff7b0993917209809ff3d94bec7e1de7188b323d9b88e0273cb71
> S2:P:00014979
> S2:F:000149794d90dc5bce4f685deed6076d00c9209bd81cef4cbdf8a4e41f0a2153
> ...
>
>
> That version loads in 50+ seconds for the 3229612 entries in it.
>
> If I flip the order of the entries so the :F: entries comes before the
> corresponding :P: entry, it loads the same number of entries in 5 - 10
> seconds.
> If I reorder the entire file so that _all_ the :F: entries for each
> section (S or S2) come before the :P: entries for that section, it loads in
> under 5 seconds again.
>
> Earlier today it was mentioned that 'the next version of the CVD' would
> fix it (when 48473 was the current version). That seems to have not been
> the case since 48474 didn't fix it. Is there a plan to fix it? Or will we
> have to live with the enormous load times for this database?
>
> --Maarten
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>


-- 
---
Dave Raynor
Talos Security Intelligence and Research Group
draynor at sourcefire.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20190306/897ff40c/attachment.html>


More information about the clamav-users mailing list