[clamav-users] ClamAV HTML RealURL DisplayURL failed
shishabert at vollbio.de
shishabert at vollbio.de
Wed Jul 29 13:54:55 UTC 2020
Hi,
what do you mean with "writing your rule"?
amavis works fine - i put the realURL in the body of mail and he alerts me. he alterted me too, when I use the the badevil-link e.g. "https[:// bad-boy-link[.com/path/to/location/" in my yara-rule and take in my mail-body with an hyperlink (realURL: "https[:// bad-boy-link[.com/path/to/location/" / displayURL: "https[:// I-am-so-innocent[.com/click-me/"). Only ClamAV do not find or does not recognize, if the link are hyperlink:
clamscan -d /var/lib/clamav/urlhaus.ndb --debug --max-filesize=0 /root/_test/BadMessages.msg 2> test.txt
LibClamAV debug: searching for unrar, user-searchpath: /usr/lib64
LibClamAV debug: unrar support loaded from /usr/lib64/libclamunrar_iface.so.9.0.4 libclamunrar_iface_so_9_0
LibClamAV debug: Initialized 0.102.2 engine
LibClamAV debug: Initializing phishcheck module
LibClamAV debug: Phishcheck: Compiling regex: ^ *(http|https|ftp:(//)?)?[0-9]{1,3}(\.[0-9]{1,3}){3}[/?:]? *$
LibClamAV debug: Phishcheck module initialized
LibClamAV debug: Bytecode initialized in interpreter mode
LibClamAV debug: Initializing engine->root[0]
LibClamAV debug: Initializing AC pattern matcher of root[0]
LibClamAV debug: cli_initroots: Initializing BM tables of root[0]
LibClamAV debug: Initializing engine->root[1]
LibClamAV debug: Initializing AC pattern matcher of root[1]
LibClamAV debug: cli_initroots: Initializing BM tables of root[1]
LibClamAV debug: Initializing engine->root[2]
...
...
...
LibClamAV debug: /var/lib/clamav/urlhaus.ndb loaded
LibClamAV debug: Loaded 155 filetype definitions
LibClamAV debug: Using filter for trie 0
LibClamAV debug: Matcher[0]: GENERIC: AC sigs: 82 (reloff: 1, absoff: 0) BM sigs: 5360 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 251
LibClamAV debug: Using filter for trie 1
LibClamAV debug: Matcher[1]: PE: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0
LibClamAV debug: Matcher[2]: OLE2: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[3]: HTML: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Using filter for trie 4
LibClamAV debug: Matcher[4]: MAIL: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[5]: GRAPHICS: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[6]: ELF: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Using filter for trie 7
LibClamAV debug: Matcher[7]: ASCII: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[8]: NOT USED: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[9]: MACH-O: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[10]: PDF: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[11]: FLASH: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[12]: JAVA: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[13]: INTERNAL: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Matcher[14]: OTHER: AC sigs: 0 (reloff: 0, absoff: 0) BM sigs: 0 (reloff: 0, absoff: 0) PCREs: 0 (reloff: 0, absoff: 0) maxpatlen 0 (ac_only mode)
LibClamAV debug: Dynamic engine configuration settings:
LibClamAV debug: --------------------------------------
LibClamAV debug: Module PE: On
LibClamAV debug: * Submodule PARITE: On
LibClamAV debug: * Submodule KRIZ: On
LibClamAV debug: * Submodule MAGISTR: On
LibClamAV debug: * Submodule POLIPOS: On
LibClamAV debug: * Submodule MD5SECT: On
LibClamAV debug: * Submodule UPX: On
LibClamAV debug: * Submodule FSG: On
LibClamAV debug: * Submodule SWIZZOR: ** Off **
LibClamAV debug: * Submodule PETITE: On
LibClamAV debug: * Submodule PESPIN: On
LibClamAV debug: * Submodule YC: On
LibClamAV debug: * Submodule WWPACK: On
LibClamAV debug: * Submodule NSPACK: On
LibClamAV debug: * Submodule MEW: On
LibClamAV debug: * Submodule UPACK: On
LibClamAV debug: * Submodule ASPACK: On
LibClamAV debug: * Submodule CATALOG: On
LibClamAV debug: * Submodule CERTS: On
LibClamAV debug: * Submodule MATCHICON: On
LibClamAV debug: * Submodule IMPTBL: On
LibClamAV debug: Module ELF: On
LibClamAV debug: Module MACHO: On
LibClamAV debug: Module ARCHIVE: On
LibClamAV debug: * Submodule RAR: On
LibClamAV debug: * Submodule ZIP: On
LibClamAV debug: * Submodule GZIP: On
LibClamAV debug: * Submodule BZIP: On
LibClamAV debug: * Submodule ARJ: On
LibClamAV debug: * Submodule SZDD: On
LibClamAV debug: * Submodule CAB: On
LibClamAV debug: * Submodule CHM: On
LibClamAV debug: * Submodule OLE2: On
LibClamAV debug: * Submodule TAR: On
LibClamAV debug: * Submodule CPIO: On
LibClamAV debug: * Submodule BINHEX: On
LibClamAV debug: * Submodule SIS: On
LibClamAV debug: * Submodule NSIS: On
LibClamAV debug: * Submodule AUTOIT: On
LibClamAV debug: * Submodule ISHIELD: On
LibClamAV debug: * Submodule 7zip: On
LibClamAV debug: * Submodule ISO9660: On
LibClamAV debug: * Submodule DMG: On
LibClamAV debug: * Submodule XAR: On
LibClamAV debug: * Submodule HFSPLUS: On
LibClamAV debug: * Submodule XZ: On
LibClamAV debug: * Submodule PASSWD: On
LibClamAV debug: * Submodule MBR: On
LibClamAV debug: * Submodule GPT: On
LibClamAV debug: * Submodule APM: On
LibClamAV debug: * Submodule EGG: On
LibClamAV debug: Module DOCUMENT: On
LibClamAV debug: * Submodule HTML: On
LibClamAV debug: * Submodule RTF: On
LibClamAV debug: * Submodule PDF: On
LibClamAV debug: * Submodule SCRIPT: On
LibClamAV debug: * Submodule HTMLSKIPRAW: On
LibClamAV debug: * Submodule JSNORM: On
LibClamAV debug: * Submodule SWF: On
LibClamAV debug: * Submodule OOXML: On
LibClamAV debug: * Submodule MSPML: On
LibClamAV debug: * Submodule HWP: On
LibClamAV debug: Module MAIL: On
LibClamAV debug: * Submodule MBOX: On
LibClamAV debug: * Submodule TNEF: On
LibClamAV debug: Module OTHER: On
LibClamAV debug: * Submodule UUENCODED: On
LibClamAV debug: * Submodule SCRENC: On
LibClamAV debug: * Submodule RIFF: On
LibClamAV debug: * Submodule JPEG: On
LibClamAV debug: * Submodule CRYPTFF: On
LibClamAV debug: * Submodule DLP: On
LibClamAV debug: * Submodule MYDOOMLOG: On
LibClamAV debug: * Submodule PREFILTERING: On
LibClamAV debug: * Submodule PDFNAMEOBJ: On
LibClamAV debug: * Submodule PRTNINTXN: On
LibClamAV debug: * Submodule LZW: On
LibClamAV debug: Module PHISHING On
LibClamAV debug: * Submodule ENGINE: On
LibClamAV debug: * Submodule ENTCONV: On
LibClamAV debug: Module BYTECODE On
LibClamAV debug: * Submodule INTERPRETER: On
LibClamAV debug: * Submodule JIT X86: On
LibClamAV debug: * Submodule JIT PPC: On
LibClamAV debug: * Submodule JIT ARM: ** Off **
LibClamAV debug: Module STATS Off
LibClamAV debug: Module PCRE On
LibClamAV debug: * Submodule SUPPORT: On
LibClamAV debug: * Submodule OPTIONS: On
LibClamAV debug: * Submodule GLOBAL: On
LibClamAV debug: pool memory used: 6.683 MB
LibClamAV debug: No bytecodes loaded, not running builtin test
LibClamAV debug: in cli_magic_scandesc (reclevel: 0/16)
LibClamAV debug: Recognized OLE2 container file
LibClamAV debug: cache_check: 93cf4c97f167a4ee6785c255f08a86ff is negative
LibClamAV debug: in cli_scanole2()
LibClamAV debug: in cli_ole2_extract()
LibClamAV debug:
LibClamAV debug: Magic: 0xd0cf11e0a1b11ae1
LibClamAV debug: CLSID: {0000-00-00-00-000000}
LibClamAV debug: Minor version: 0x3e
LibClamAV debug: DLL version: 0x3
LibClamAV debug: Byte Order: -2
LibClamAV debug: Big Block Size: 9
LibClamAV debug: Small Block Size: 6
LibClamAV debug: BAT count: 1
LibClamAV debug: Prop start: 2
LibClamAV debug: SBAT cutoff: 4096
LibClamAV debug: SBat start: 23
LibClamAV debug: SBat block count: 2
LibClamAV debug: XBat start: -2
LibClamAV debug: XBat block count: 0
LibClamAV debug:
LibClamAV debug: Max block number: 592
LibClamAV debug: OLE2: no VBA projects found
LibClamAV debug: OLE2: __substg1.0_1035001f [file] b size:0x00000058 flags:0x00000000
LibClamAV debug: OLE2 [handler_otf]: Dumping '__substg1.0_1035001f' to '/tmp/clamav-43c3c2403f7dd247e85e9e8c60f9b18a.tmp'
LibClamAV debug: in cli_magic_scandesc (reclevel: 1/16)
LibClamAV debug: Recognized UTF-16BE character data
LibClamAV debug: cache_check: 62ce5a3c9cb94c4046b38f0e1b890d7a is negative
LibClamAV debug: in cli_check_mydoom_log()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: in cli_scanscript()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: cli_magic_scandesc: returning 0 at line 3202
LibClamAV debug: cache_add: 62ce5a3c9cb94c4046b38f0e1b890d7a (level 0)
LibClamAV debug: OLE2: __substg1.0_5d01001f [file] b size:0x00000028 flags:0x00000000
LibClamAV debug: OLE2 [handler_otf]: Dumping '__substg1.0_5d01001f' to '/tmp/clamav-6c6a6e130a904a0c83472e456724457e.tmp'
LibClamAV debug: in cli_magic_scandesc (reclevel: 1/16)
LibClamAV debug: Recognized UTF-16BE character data
LibClamAV debug: cache_check: 6cda96ff40c2bde75aa64323d29b29d0 is negative
LibClamAV debug: in cli_check_mydoom_log()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: in cli_scanscript()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: cli_magic_scandesc: returning 0 at line 3202
LibClamAV debug: cache_add: 6cda96ff40c2bde75aa64323d29b29d0 (level 0)
LibClamAV debug: OLE2: __substg1.0_8005001f [file] b size:0x000000fe flags:0x00000000
LibClamAV debug: OLE2 [handler_otf]: Dumping '__substg1.0_8005001f' to '/tmp/clamav-148939a3f5107554c19fa07d92d7ecfd.tmp'
LibClamAV debug: in cli_magic_scandesc (reclevel: 1/16)
LibClamAV debug: Recognized UTF-16BE character data
LibClamAV debug: cache_check: 9da80f4edffef7fd09cbbc0b5c2c4456 is negative
LibClamAV debug: in cli_check_mydoom_log()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: in cli_scanscript()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: cli_magic_scandesc: returning 0 at line 3202
LibClamAV debug: cache_add: 9da80f4edffef7fd09cbbc0b5c2c4456 (level 0)
LibClamAV debug: OLE2: __substg1.0_800c001f [file] b size:0x00000004 flags:0x00000000
LibClamAV debug: OLE2 [handler_otf]: Dumping '__substg1.0_800c001f' to '/tmp/clamav-5bc7a7e6cc75d3fd3c4581ac650c0dad.tmp'
...
...
...
LibClamAV debug: OLE2 [handler_otf]: Dumping '__substg1.0_10030102' to '/tmp/clamav-478bfa13b0733061d8f989771e12de15.tmp'
LibClamAV debug: in cli_magic_scandesc (reclevel: 1/16)
LibClamAV debug: Recognized UTF-16BE character data
LibClamAV debug: cache_check: 4e8515af492d75f968653ed67546d706 is negative
LibClamAV debug: in cli_check_mydoom_log()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: in cli_scanscript()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: cli_magic_scandesc: returning 0 at line 3202
LibClamAV debug: cache_add: 4e8515af492d75f968653ed67546d706 (level 0)
LibClamAV debug: OLE2: __substg1.0_00020102 [file] b size:0x00000060 flags:0x00000000
LibClamAV debug: OLE2 [handler_otf]: Dumping '__substg1.0_00020102' to '/tmp/clamav-11e2843eef1940d504ace2cc3d3e0e11.tmp'
LibClamAV debug: in cli_magic_scandesc (reclevel: 1/16)
LibClamAV debug: Recognized binary data
LibClamAV debug: cache_check: 610f92af7c00ed29bb77465b4714c36d is negative
LibClamAV debug: in cli_check_mydoom_log()
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: cli_magic_scandesc: returning 0 at line 3202
LibClamAV debug: cache_add: 610f92af7c00ed29bb77465b4714c36d (level 0)
LibClamAV debug: Matched signature for file type HTML data at 20288
LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
LibClamAV debug: cli_magic_scandesc: returning 0 at line 3202
LibClamAV debug: cache_add: 93cf4c97f167a4ee6785c255f08a86ff (level 0)
LibClamAV debug: Cleaning up phishcheck
LibClamAV debug: Freeing phishcheck struct
LibClamAV debug: Phishcheck cleaned up
the following plugins are activated:
======================================
Jul 29 15:30:58 clamd[18529]: Archive support enabled.
Jul 29 15:30:58 clamd[18529]: AlertExceedsMax heuristic detection disabled.
Jul 29 15:30:58 clamd[18529]: Heuristic alerts enabled.
Jul 29 15:30:58 clamd[18529]: Portable Executable support enabled.
Jul 29 15:30:58 clamd[18529]: ELF support enabled.
Jul 29 15:30:58 clamd[18529]: Mail files support enabled.
Jul 29 15:30:58 clamd[18529]: OLE2 support enabled.
Jul 29 15:30:58 clamd[18529]: PDF support enabled.
Jul 29 15:30:58 clamd[18529]: SWF support enabled.
Jul 29 15:30:58 clamd[18529]: HTML support enabled.
Jul 29 15:30:58 clamd[18529]: XMLDOCS support enabled.
Jul 29 15:30:58 clamd[18529]: HWP3 support enabled.
Jul 29 15:30:58 clamd[18529]: Heuristic: precedence enabled
Jul 29 15:30:58 clamd[18529]: Self checking every 600 seconds.
My Amavisd part for clamav:
======================================
@virus_name_to_spam_score_maps = (new_RE(
[ qr'^Phishing\.' => 6.1 ],
[ qr'^(Heuristics\.)?Phishing\.' => 6.1 ],
[ qr'^Structured\.(SSN|CreditCardNumber)\b' => 6.1 ],
[ qr'^(?:Email|HTML|Sanesecurity)\.(?:Phishing|SpearL?)\.'i => 6.1 ],
[ qr'^(?:Email|HTML|Sanesecurity)\.(?:Spam|Scam)[a-z0-9]?\.'i => 6.1 ],
[ qr'^Sanesecurity\.(Malware|Rogue|Badmacro|Trojan)\.' => undef ],
[ qr'^Email\.Spam.*-SecuriteInfo\.com(\.|\z)' => 6.1 ],
[ qr'^SecuriteInfo\.com\.Spam\.' => 6.1 ],
[ qr'^winnow\.(?:botnets?|phish|complex|mailer)\.'x => 6.1 ],
[ qr'^winnow\.spam(?:domain)?\.'x => 6.1 ],
[ qr'^winnow\.(?:malware|trojan|compromised)\.'x => undef ],
[ qr'^winnow\.'x => 6.1 ],
[ qr'^PhishTank\.Phishing\.' => 6.1 ],
[ qr'^Bofhland\.Malware\.' => undef ],
[ qr'^Porcupine\.(Malware|JS|Java|Win32|MSIL|VBS)\.' => undef ],
[ qr'^Porcupine\.' => 6.1 ],
[ qr'^lw\.' => 6.1 ],
[ qr'^YARA\.invalid_xref_numbers\.' => 3.2 ],
[ qr'^YARA\.multiple_filtering\.' => 3.2 ],
[ qr'^YARA\.suspicious_version\.' => 3.2 ],
[ qr'^URLhaus\.' => undef ],
[ qr'^MBL_' => 5.8 ]
));
I don't know why! :/
BR, Bert
> Gesendet: Mittwoch, 29. Juli 2020 um 14:33 Uhr
> Von: "Joel Esler (jesler) via clamav-users" <clamav-users at lists.clamav.net>
> An: "ClamAV users ML" <clamav-users at lists.clamav.net>
> Cc: "Joel Esler (jesler)" <jesler at cisco.com>
> Betreff: Re: [clamav-users] ClamAV HTML RealURL DisplayURL failed
>
> Are you writing your rule to detect the correct file type?
>
> Sent from my iPad
>
> > On Jul 29, 2020, at 06:02, shishabert at vollbio.de wrote:
> >
> > hi @ all,
> >
> > i use postfix, amavisd and clamav with urlhaus ndb (for ClamAV) sig from urlhaus.abuse.ch. if i send or receive a mail with a hyperlink - realURL/ displayURL like :
> >
> > ...
> > ...
> > <a href="https:// example-from-urlhaus.[com/link/to/location/">https:// foo-bar-anything-blubb.[com/happy-malware-fakename</a><o:p></o:p></p>
> > ...
> > ...
> >
> > clamav does not recognize this. but, if I place the link directly in the mail body (HTML format) clamav recognizes this:
> >
> > clamd[25845]: /var/amavis/tmp/amavis-20200729T082557-25999-Hy3LWJ3x/parts/p004: URLhaus.421252.UNOFFICIAL FOUND
> >
> > And when i create a yara rule with the link to urlhaus.abuse.ch it detects the badevil-url link without problems.
> > for example:
> >
> > ...
> > LibClamAV debug: FP SIGNATURE: cef114bc2adc4caeaf51f716ba3c1611:923:YARA.spam_subject.UNOFFICIAL
> > LibClamAV debug: hashtab: Freeing hashset, elements: 0, capacity: 0
> > LibClamAV debug: YARA.spam_subject.UNOFFICIAL found
> >
> >
> > you can tell what I'm doing wrong?
> >
> > BR, Bert
> >
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users at lists.clamav.net
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users at lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
More information about the clamav-users
mailing list