On Sat, 10 Oct 2020, Robert Kudyba wrote:

> ... next time it happens I can try some of these:
> ...

But put some logging in place before it does, so you get as precise a
timeline as you can.

> Here's what the -i option returns:
> ...
> Loading config: /etc/clamav-unofficial-sigs/master.conf
> Loading config: /etc/clamav-unofficial-sigs/os.conf
> Loading config: /etc/clamav-unofficial-sigs/user.conf

I take it you've examined these files for clues?  And the systemd unit
files etc.?

Indeed and here we are 9 months later and the problem is back. I can see this happened after Jul 3 at 4:22 AM:
Jul 03 04:22:22 Checking for updated interServer database file: interservertopline.db

Jul 03 04:22:22 No updated interServer interservertopline.db database file

Jul 03 04:22:22 No interServer database file updates

Jul 03 04:22:22 MalwarePatrol Database File Updates

Jul 03 04:22:22 24 hours have not yet elapsed since the last malwarepatrol update check

Jul 03 04:22:22 No update check was performed at this time

Jul 03 04:22:22 Next check will be performed in approximately 6 hour(s), 53 minute(s)

Jul 03 04:22:22 URLhaus Database File Updates

Jul 03 04:22:22 Checking for urlhaus updates...

Jul 03 04:22:22 Checking for updated urlhaus database file: urlhaus.ndb

Jul 03 04:22:22 WARNING: Failed connection to https://urlhaus.abuse.ch/downloads - SKIPPED urlhaus urlhaus.ndb update

Jul 03 04:22:22 No updated urlhaus urlhaus.ndb database file

Jul 03 04:22:22 No urlhaus database file updates

Jul 03 04:22:22 Yara-Rules Database File Updates

Jul 03 04:22:22 24 hours have not yet elapsed since the last yararulesproject update check

Jul 03 04:22:22 No update check was performed at this time

Jul 03 04:22:22 Next check will be performed in approximately 6 hour(s), 53 minute(s)

Jul 03 04:22:22 Update(s) detected, reloading ClamAV databases

Jul 03 04:22:22 ClamAV databases reloading

Jul 03 04:22:22 Issue tracker : https://github.com/extremeshok/clamav-unofficial-sigs/issues

Jul 03 04:22:22       Powered By https://eXtremeSHOK.com      

Jul 03 05:14:01 ERROR: clam database directory (clam_dbs) not writable /var/lib/clamav


 ps -auwx|grep clam

clamav   1533123  0.0  1.2 2783400 1678272 ?     Ssl  Jul03   7:13 /usr/sbin/clamd -c /etc/clamd.d/scan.conf

clamilt  1533191  0.0  0.0 1053352 3616 ?        Ssl  Jul03   0:05 /usr/sbin/clamav-milter -c /etc/mail/clamav-milter.conf

clamav   1533209  0.0  0.0  28268 12480 ?        Ss   Jul03   0:00 /usr/bin/freshclam -d --foreground=true


ls -ld /var/lib/clamav

drwxr-xr-x. 4 clamupdate clamupdate 8192 Jul  3 04:46 /var/lib/clamav


and these 3 files have their owner changed but note the old date timestamp: 

-rw-r--r--  1 clamupdate clamupdate    293670 Apr  8 06:32 bytecode.cvd

-rw-r--r--  1 clamupdate clamupdate 107169718 Jun 22 18:06 daily.cvd

-rw-r--r--  1 clamupdate clamupdate 117859675 Nov 25  2019 main.cvd


grep clamupdate /etc/clam*/*

/etc/clamav-unofficial-sigs/os.conf:#clam_user="clamupdate"

/etc/clamav-unofficial-sigs/os.conf:#clam_group="clamupdate"


status clamav-freshclam.service 

clamav-freshclam.service - ClamAV virus database updater

     Loaded: loaded (/usr/lib/systemd/system/clamav-freshclam.service; enabled; vendor preset: disabled)

     Active: active (running) since Sat 2021-07-03 04:46:13 EDT; 1 weeks 1 days ago

       Docs: man:freshclam(1)

             man:freshclam.conf(5)

             https://www.clamav.net/documents

   Main PID: 1533209 (freshclam)

      Tasks: 1 (limit: 154192)

     Memory: 1.7M

     CGroup: /system.slice/clamav-freshclam.service

             └─1533209 /usr/bin/freshclam -d --foreground=true


Jul 11 20:46:13 ourserver.edu freshclam[1533209]: ERROR: Can't create temporary directory /var/lib/clamav/tmp.92f6163053

Jul 11 20:46:13 ourserver.edu freshclam[1533209]: Hint: The database directory must be writable for UID 985 or GID 981

Jul 11 20:46:13 ourserver.edu freshclam[1533209]: ERROR: Update failed.

Jul 11 20:46:13 ourserver.edu freshclam[1533209]: Received signal: wake up

Jul 11 20:46:13 ourserver.edu freshclam[1533209]: ClamAV update process started at Sun Jul 11 20:46:13 2021

Jul 11 20:46:13 ourserver.edu freshclam[1533209]: DNS record is older than 3 hours.

Jul 11 20:46:13 ourserver.edu freshclam[1533209]: Can't create temporary directory /var/lib/clamav/tmp.92f6163053

Jul 11 20:46:13 ourserver.edu freshclam[1533209]: Hint: The database directory must be writable for UID 985 or GID 981

Jul 11 20:46:13 ourserver.edu freshclam[1533209]: Update failed.

Jul 11 20:46:13 ourserver.edu freshclam[1533209]: --------------------------------------


cat /usr/lib/systemd/system/clamav-freshclam.service 

[Unit]

Description=ClamAV virus database updater

Documentation=man:freshclam(1) man:freshclam.conf(5) https://www.clamav.net/documents

# If user wants it run from cron, don't start the daemon.

ConditionPathExists=!/etc/cron.d/clamav-update

Wants=network-online.target

After=network-online.target


[Service]

ExecStart=/usr/bin/freshclam -d --foreground=true


[Install]

WantedBy=multi-user.target


systemctl status clamav-unofficial-sigs.service 

● clamav-unofficial-sigs.service - Clamav Unofficial Sigs Update service

     Loaded: loaded (/etc/systemd/system/clamav-unofficial-sigs.service; static)

     Active: inactive (dead)

       Docs: man:clamav-unofficial-sigs(8)

(base) [root@ourserver ~]# systemctl status clamav-unofficial-sigs.timer 

● clamav-unofficial-sigs.timer - Clamav Unofficial Sigs Update timer

     Loaded: loaded (/etc/systemd/system/clamav-unofficial-sigs.timer; disabled; vendor preset: disabled)

     Active: inactive (dead)

    Trigger: n/a

   Triggers: ● clamav-unofficial-sigs.service

       Docs: man:clamav-unofficial-sigs(8)


in /etc/cron.d/clamav-unofficial-sigs we have:


14 * * * *  clamav [ -x /usr/local/sbin/clamav-unofficial-sigs.sh ] && /usr/bin/bash /usr/local/sbin/clamav-unofficial-sigs.sh


Is this a clue in the system logs? UID 985 = clamav


Jul  3 04:22:32 ourserver systemd[1]: Stopping User Manager for UID 985...

Jul  3 04:22:32 ourserver systemd[1519673]: Stopped target Main User Target.

Jul  3 04:22:32 ourserver systemd[1519673]: Stopped target Basic System.

Jul  3 04:22:32 ourserver systemd[1519673]: Stopped target Paths.


grep 985 /etc/passwd

clamav:x:985:981::/var/run/clamav:/sbin/nologin