[clamav-users] Problem with new safebrowsing file

Maarten Broekman maarten.broekman at gmail.com
Wed Mar 6 17:55:34 EST 2019


I have 48472 and 48473. The 48474 I got was the gdb file that was
downloaded as part of the cdiff. The freshclam process hung after
downloading though.  The order of the 48474 gdb file was no different from
the order of the 48473 file.

Freshclam gets this far before hanging after the download. The gdb file
listed there has the same format.

Wed Mar  6 16:50:46 2019 -> *main.cvd version from DNS: 58
Wed Mar  6 16:50:46 2019 -> main.cvd is up to date (version: 58, sigs:
4566249, f-level: 60, builder: sigmgr)
Wed Mar  6 16:50:46 2019 -> *daily.cvd version from DNS: 25380
Wed Mar  6 16:50:46 2019 -> daily.cvd is up to date (version: 25380, sigs:
1503528, f-level: 63, builder: raynman)
Wed Mar  6 16:50:46 2019 -> *safebrowsing.cvd version from DNS: 48474
LibClamAV debug: in cli_untgz()
LibClamAV debug: cli_untgz: Unpacking
/home/logins/mbroekman/analysis/tmp/clamav-317041d4b9d853e83b60005464dd098c.tmp/clamav-b4a94beaae2191e11c7805c6e49be7e6.tmp/COPYING
LibClamAV debug: cli_untgz: Unpacking
/home/logins/mbroekman/analysis/tmp/clamav-317041d4b9d853e83b60005464dd098c.tmp/clamav-b4a94beaae2191e11c7805c6e49be7e6.tmp/
safebrowsing.info
LibClamAV debug: cli_untgz: Unpacking
/home/logins/mbroekman/analysis/tmp/clamav-317041d4b9d853e83b60005464dd098c.tmp/clamav-b4a94beaae2191e11c7805c6e49be7e6.tmp/safebrowsing.gdb
LibClamAV debug: in cli_untgz_cleanup()
Wed Mar  6 16:50:49 2019 -> *Retrieving
http://db.US.clamav.net/safebrowsing-48474.cdiff
Wed Mar  6 16:50:49 2019 -> nonblock_connect: connect(): fd=4 errno=101:
Network is unreachable
Wed Mar  6 16:50:49 2019 -> Can't connect to port 80 of host
db.US.clamav.net (IP: 2606:4700::6810:da54)
Wed Mar  6 16:50:49 2019 -> *Trying to download
http://db.US.clamav.net/safebrowsing-48474.cdiff (IP: 104.16.219.84)
Wed Mar  6 16:50:49 2019 -> Downloading safebrowsing-48474.cdiff [100%]


The last time I ran freshclam, I was stuck at 100% on the download for 40
minutes before I killed the process.

The info file in the tmpdir shows:

ClamAV-VDB:06 Mar 2019 13-24 -0500:48474:3232286:63:X:X:google:1551896655
safebrowsing.gdb:132636452:7f6645b8d865de3992be1ad5de215afd848acee4c021eed4818fdb760f76b57e
DSIG:NxsTJGIb7EQ9e71CjIH2QJYzp+BhrH0qK1Mb0Ef5BQfO5WZnm8qZSqj/y6vstcjAOUfWwLG8ba3RemesF+KxIuk/HMkDgRCJep+shVvz8nAccajvbBN1ZnmpTkf1T0QgTsDbuBK9cTItdlQWupKfuiV1aKKdF1jSLvtRJU4zoZl+B3/qgIAPi7sqmkh8W5qKplYdsICdfmDLxK5dDwCkGmdtXZol5pHHXTQb1/LJqml8SORrFydkYizuVl07/uuc332dk5Uk1NfZrDj94wG0dIIloWiwfPzj563Vl5e7GvCvCdMR1Gfq3EGYZGSPftR7a/K7TashvsoWP2Uma0Fq/





On Wed, Mar 6, 2019 at 5:47 PM David Raynor <draynor at sourcefire.com> wrote:

> That's strange, the 48474 I have should have the sorting changed and has
> the improved loading time we're talking about.
>
> $ sigtool --info safebrowsing.cvd
> File: safebrowsing.cvd
> Build time: 06 Mar 2019 13:24 -0500
> Version: 48474
> Signatures: 3232286
> Functionality level: 63
> Builder: google
> MD5: 70c61f41e52b5a2134ff7e272f5a6df1
>
> SHA256 (safebrowsing.gdb) =
> 7f6645b8d865de3992be1ad5de215afd848acee4c021eed4818fdb760f76b57e
>
> Something must be different.
>
> Dave R.
>
> On Wed, Mar 6, 2019 at 5:39 PM Maarten Broekman via clamav-users <
> clamav-users at lists.clamav.net> wrote:
>
>> The new safebrowsing cvd (starting with version 48473) seems to be sorted
>> in a way that increases the load time of that file by several orders of
>> magnitude.
>>
>> I have a previous version from February where the entries in the gdb
>> section are sorted like this:
>>
>> S2:F:0000917787cff7b0993917209809ff3d94bec7e1de7188b323d9b88e0273cb71
>> S2:F:000149794d90dc5bce4f685deed6076d00c9209bd81cef4cbdf8a4e41f0a2153
>> S2:F:00042c895c912fd567afa35450cfe5d321d0d68eb3833156925c4e27d2c29aa2
>> S2:F:0006d4dcb0d939d725e676a9e68aaeb303e04478e6861d2a77469d1b6a0a0f7d
>> S2:F:0007bf7c1808d12177f0ae90d336d60c5a7a3d89703806955b75c56f898dd919
>> ...
>> S2:P:00009177
>> S2:P:00014979
>> S2:P:00042c89
>> S2:P:0006d4dc
>> S2:P:0007bf7c
>> ...
>> S:F:00000860493997b798861956e06d3d3606f82384259b971bb922f94f886a4b55
>> S:F:00000bddafae162a7a2f1249b3b38c8e4b6d3cb8bf0c30c26cc354ebcba16b37
>> S:F:000046cad35fbecbcc8dd4ebb244bd08aa6dbf1078279115c82f8e21b2cf8478
>> S:F:0000684200da7b11f38a6f4719bda4ec6c6ae8b2be1f7e12a16605b2d3a5d490
>> S:F:000072f3f33e47a2f97b8711d240267462aa3f0a5f8130845b119a2ad3798292
>> ...
>> S:P:00000860
>> S:P:00000bdd
>> S:P:000046ca
>> S:P:00006842
>> S:P:000072f3
>>
>>
>> That loads into clamd (and clamscan) in under 5 seconds for the 3041760
>> entries in it.
>>
>> Version 48473 and 48474 are sorted like this:
>>
>> S2:P:00009177
>> S2:F:0000917787cff7b0993917209809ff3d94bec7e1de7188b323d9b88e0273cb71
>> S2:P:00014979
>> S2:F:000149794d90dc5bce4f685deed6076d00c9209bd81cef4cbdf8a4e41f0a2153
>> ...
>>
>>
>> That version loads in 50+ seconds for the 3229612 entries in it.
>>
>> If I flip the order of the entries so the :F: entries comes before the
>> corresponding :P: entry, it loads the same number of entries in 5 - 10
>> seconds.
>> If I reorder the entire file so that _all_ the :F: entries for each
>> section (S or S2) come before the :P: entries for that section, it loads in
>> under 5 seconds again.
>>
>> Earlier today it was mentioned that 'the next version of the CVD' would
>> fix it (when 48473 was the current version). That seems to have not been
>> the case since 48474 didn't fix it. Is there a plan to fix it? Or will we
>> have to live with the enormous load times for this database?
>>
>> --Maarten
>>
>>
>>
>> _______________________________________________
>>
>> clamav-users mailing list
>> clamav-users at lists.clamav.net
>> https://lists.clamav.net/mailman/listinfo/clamav-users
>>
>>
>> Help us build a comprehensive ClamAV guide:
>> https://github.com/vrtadmin/clamav-faq
>>
>> http://www.clamav.net/contact.html#ml
>>
>
>
> --
> ---
> Dave Raynor
> Talos Security Intelligence and Research Group
> draynor at sourcefire.com
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clamav.net/pipermail/clamav-users/attachments/20190306/2c68e27c/attachment.html>


More information about the clamav-users mailing list