[clamav-users] reloading database problem

[Matus UHLAR - fantomas]

>On Wed, 9 Feb 2022, Matus UHLAR - fantomas wrote:
>
>>I have clamav 0.103.5 installed on debian 11 and I'm getting too often
>>errors when reloading database.
>>
>>looking back this problem started appearing on:
>>
>>Mon May 10 11:51:15 2021 -> Database correctly reloaded (12721518 signatures)
>>Mon May 10 12:48:11 2021 -> ERROR: reload_th: Database load failed: Malformed database
>>...

On 09.02.22 09:44, G.W. Haywood via clamav-users wrote:
>What a lot of signatures!  I'm at around 8.8 million at the moment,
>with about 45 additional third-party databases and yara rule sets.

I think most of it comes from securiteinfo.com feed, which I have 
subscribed into. I have this machine for personal use.

it seems their signatures are the most commonly catched:

% zgrep -Fih FOUND `ls -1tr clamav.log*` | awk '$8 == "(deleted):" {print $9;next} {print $8}' | cut -f1 -d. | sort | uniq -c|sort -nr
      84 SecuriteInfo
      62 Porcupine
      32 Sanesecurity
       2 PhishTank
       1 Bofhland

(there may be duplicates so the real difference may be smaller)
  
>>this machine has 4G of RAM and some swap, clamd currently eats ~1.5 GB ...
>
>With 8.8M sigs on ARM7 64 bit with 4G RAM I'm using about 1.2GB of
>resident memory and concurrent reloads give no trouble.  There were
>some 'malformed' bleatings in the log back at the end of June - early
>July, but I think that was a real database problem which was promptly
>fixed.  Nothing at all since then.
>
>>I wonder if this problem may be caused by i386 architecture with 3GB limit ...
>>Does clamd reload signature database in the same process?
>
>It's a very long time since I ran ClamAV on i386 so I've no experience
>to offer.  If your suspicion is correct it might be a problem specific
>to the machine:
>
>https://en.wikipedia.org/wiki/3_GB_barrier

yes, this is what I'm guessing.
I'm just curious if someone can confirm this or I have to try.
so far I was lazy to convert this machine (or at least part of it) to 
64-bit. 64-bit kernel should help to move the barrier to 4G.

>There's a configuration option to avoid the doubled memory usage on a
>database reload, look in the configuration file for clamd for the
>'ConcurrentDatabaseReload' directive.  Be aware of the issues, you
>might not want to pause scanning during reloads.

I know of this feature, just wanted to avoid it.

>>is the "Malformed database" just incorrect error code for this case?
>
>It's not impossible.  One of the most valuable lessons I learned early
>in my career was not to put too much faith in the error messages given
>by most computer software.  Sometimes I will recompile an executable
>with a bunch extra error messages when I wonder if I understand what's
>going on (the ClamAV error handling is generally pretty well organized
>which makes that easy).  But if you stress things enough you're always
>going to find corner cases.

-- 
Matus UHLAR - fantomas, uhlar at fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
10 GOTO 10 : REM (C) Bill Gates 1998, All Rights Reserved!