[clamav-users] reloading database problem

Sun Feb 13 10:14:30 UTC 2022

>>On Wed, 9 Feb 2022, Matus UHLAR - fantomas wrote:
>>>I have clamav 0.103.5 installed on debian 11 and I'm getting too often
>>>errors when reloading database.
>>>
>>>looking back this problem started appearing on:
>>>
>>>Mon May 10 11:51:15 2021 -> Database correctly reloaded (12721518 signatures)
>>>Mon May 10 12:48:11 2021 -> ERROR: reload_th: Database load failed: Malformed database
[...]
>>>this machine has 4G of RAM and some swap, clamd currently eats ~1.5 GB ...

>>>I wonder if this problem may be caused by i386 architecture with 3GB limit ...
>>>Does clamd reload signature database in the same process?

>On 09.02.22 09:44, G.W. Haywood via clamav-users wrote:
>>It's a very long time since I ran ClamAV on i386 so I've no experience
>>to offer.  If your suspicion is correct it might be a problem specific
>>to the machine:
>>
>>https://en.wikipedia.org/wiki/3_GB_barrier

On 10.02.22 09:58, Matus UHLAR - fantomas wrote:
>yes, this is what I'm guessing.
>I'm just curious if someone can confirm this or I have to try.
>so far I was lazy to convert this machine (or at least part of it) to 
>64-bit. 64-bit kernel should help to move the barrier to 4G.

I have rebooted into 64-bit kernel, without changing any installed software.
looks like database updates work flawlessly since:

Fri Feb 11 19:52:38 2022 -> SelfCheck: Database modification detected. Forcing reload.
Fri Feb 11 19:53:03 2022 -> ERROR: reload_th: Database load failed: Can't allocate memory
Fri Feb 11 19:53:04 2022 -> WARNING: Database reload failed, keeping the previous instance
Fri Feb 11 20:42:57 2022 -> +++ Started at Fri Feb 11 20:42:57 2022
Fri Feb 11 20:42:57 2022 -> Not loading PUA signatures.
Fri Feb 11 20:43:28 2022 -> Loaded 12726414 signatures.
Fri Feb 11 20:49:16 2022 -> Database correctly reloaded (12726430 signatures)
Fri Feb 11 20:49:16 2022 -> Activating the newly loaded database...
Fri Feb 11 21:54:23 2022 -> Database correctly reloaded (12726435 signatures)
Fri Feb 11 21:54:23 2022 -> Activating the newly loaded database...
Fri Feb 11 22:49:08 2022 -> SelfCheck: Database modification detected. Forcing reload.
Fri Feb 11 22:49:45 2022 -> Database correctly reloaded (12726434 signatures)
Fri Feb 11 22:49:45 2022 -> Activating the newly loaded database...

So the 3GB barrier applies to clamav (no wonder) when reloading signatures.
- unlike other SW, no new clamd instance after reload.

>>There's a configuration option to avoid the doubled memory usage on a
>>database reload, look in the configuration file for clamd for the
>>'ConcurrentDatabaseReload' directive.  Be aware of the issues, you
>>might not want to pause scanning during reloads.
>
>I know of this feature, just wanted to avoid it.

even my swap usage is lower, which is a good thing. 

I'm going to activate zswap again.  Before this change, my machine was 
running quite slowly, apparently because of excessive swapping due to 
repeated attempts to reload signature.

I have learnt something...

>>What a lot of signatures!  I'm at around 8.8 million at the moment,
>>with about 45 additional third-party databases and yara rule sets.

>On Thu, 10 Feb 2022, Matus UHLAR - fantomas wrote:
>>I think most of it comes from securiteinfo.com feed, which I have 
>>subscribed into. I have this machine for personal use.
>>
>>it seems their signatures are the most commonly catched:
>>
>>% zgrep -Fih FOUND `ls -1tr clamav.log*` | awk ...
>>    84 SecuriteInfo
>>    62 Porcupine
>>    32 Sanesecurity
[...]
>>(there may be duplicates so the real difference may be smaller)

On 10.02.22 09:38, G.W. Haywood via clamav-users wrote:
>That's a bit odd.  You seem to be getting roughly twice the hits from
>Porcupine that you get from Sansecurity, and over here it's the other
>way around although the difference is smaller.  We see about 50%-60%
>more from Sanesecurity than from Porcupine, 85 and 55 respectively to
>date in February.  In fact my Yara rules catch many more than that, I
>wonder if they catch more of what Porcupine would have caught and your
>SecuriteInfo sigs catch more of what Sanesecurity would have caught.

that's what I meant by duplicates.

>I've looked into telling ClamAV to report all the matches it can find
>instead of just the first, but actually doing that hasn't yet reached
>the top of this 'in' tray.  I'll stop.  A fellow could go nuts.

this could eliminate many duplicates, which could help us quite a bit.

-- 
Matus UHLAR - fantomas, uhlar at fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
42.7 percent of all statistics are made up on the spot.