However I still get these errors in sendmail:
Milter: data, reject=451 4.3.2 Please try again later

the syslog entry should give us more information.

Jul 23 11:45:33 storm systemd[1]: clamd@scan.service: Main process exited, code=killed, status=6/ABRT
Jul 23 11:45:33 storm systemd[1]: clamd@scan.service: Failed with result 'signal'.
Jul 23 11:45:33 storm systemd[1]: clamd@scan.service: Service hold-off time over, scheduling restart.
Jul 23 11:45:33 storm systemd[1]: clamd@scan.service: Scheduled restart job, restart counter is at  4.
Jul 23 11:45:33 storm systemd[1]: Stopped Generic clamav scanner daemon.
Jul 23 11:45:33 storm systemd[1]: Starting Generic clamav scanner daemon...
Jul 23 11:45:39 storm clamd[22351]: LibClamAV Error: yyerror(): /var/lib/clamav/packer.yar line 82 undefined identifier "pe"

[... snip]

Jul 23 11:46:48 storm systemd-journald[623]: Suppressed 418 messages from clamd@scan.service
Jul 23 11:46:48 storm clamd[22351]: LibClamAV Error: yyerror(): /var/lib/clamav/maldoc_somerules.yar line 245 undefined identifier "uint32be"
Jul 23 11:46:48 storm clamd[22351]: LibClamAV Warning: cli_loadyara: failed to parse or load 1 yara rules from file /var/lib/clamav/maldoc_somerules.yar, successfully loaded 15 rules.
Jul 23 11:46:55 storm systemd[1]: Started Generic clamav scanner daemon.
 

The sendmail.mc ClamAV line looks like this:
INPUT_MAIL_FILTER(`clamav-milter',`S=local:/var/run/clamav-milter/clamav-milter.socket,F=T,T=S:4m;R:4m;E:10m')dnl

Some relevant results from clamconf:

ClamdSocket = "unix:/var/run/clamd.scan/clamd.sock"
MilterSocket = "/var/run/clamav-milter/clamav-milter.socket"

note that both sendmail and clamav-milter need read/write access to the socket as
long as read/execute access to the directory (to access the socket).

I believe you mean "as well as"? Here are the permissions:

drwx--x---  2 clamilt        clamilt          60 Jul 17 15:49 clamav-milter
drwx--x---  2 clamscan       clamscan         80 Jul 17 15:49 clamd.scan

srw-r--r-- 1 clamilt virusgroup 0 Jul 17 15:49 clamav-milter.socket

-rw-rw-r-- 1 clamscan clamscan 4 Jul 17 15:49 clamd.pid
srw-rw-rw- 1 clamscan clamscan 0 Jul 17 15:49 clamd.sock