The new safebrowsing cvd (starting with version 48473) seems to be sorted in a way that increases the load time of that file by several orders of magnitude.

I have a previous version from February where the entries in the gdb section are sorted like this:

S2:F:0000917787cff7b0993917209809ff3d94bec7e1de7188b323d9b88e0273cb71

S2:F:000149794d90dc5bce4f685deed6076d00c9209bd81cef4cbdf8a4e41f0a2153

S2:F:00042c895c912fd567afa35450cfe5d321d0d68eb3833156925c4e27d2c29aa2

S2:F:0006d4dcb0d939d725e676a9e68aaeb303e04478e6861d2a77469d1b6a0a0f7d

S2:F:0007bf7c1808d12177f0ae90d336d60c5a7a3d89703806955b75c56f898dd919

...

S2:P:00009177

S2:P:00014979

S2:P:00042c89

S2:P:0006d4dc

S2:P:0007bf7c

...

S:F:00000860493997b798861956e06d3d3606f82384259b971bb922f94f886a4b55

S:F:00000bddafae162a7a2f1249b3b38c8e4b6d3cb8bf0c30c26cc354ebcba16b37

S:F:000046cad35fbecbcc8dd4ebb244bd08aa6dbf1078279115c82f8e21b2cf8478

S:F:0000684200da7b11f38a6f4719bda4ec6c6ae8b2be1f7e12a16605b2d3a5d490

S:F:000072f3f33e47a2f97b8711d240267462aa3f0a5f8130845b119a2ad3798292

...

S:P:00000860

S:P:00000bdd

S:P:000046ca

S:P:00006842

S:P:000072f3

That loads into clamd (and clamscan) in under 5 seconds for the 3041760 entries in it.

Version 48473 and 48474 are sorted like this:

S2:P:00009177

S2:F:0000917787cff7b0993917209809ff3d94bec7e1de7188b323d9b88e0273cb71

S2:P:00014979

S2:F:000149794d90dc5bce4f685deed6076d00c9209bd81cef4cbdf8a4e41f0a2153

...

That version loads in 50+ seconds for the 3229612 entries in it.

If I flip the order of the entries so the :F: entries comes before the corresponding :P: entry, it loads the same number of entries in 5 - 10 seconds.

If I reorder the entire file so that _all_ the :F: entries for each section (S or S2) come before the :P: entries for that section, it loads in under 5 seconds again.

Earlier today it was mentioned that 'the next version of the CVD' would fix it (when 48473 was the current version). That seems to have not been the case since 48474 didn't fix it. Is there a plan to fix it? Or will we have to live with the enormous load times for this database?

--Maarten

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml