[clamav-users] Latest report on update "delays"

Mon Oct 22 00:15:51 UTC 2018

You should abandon the notion of first time perfect with these kinds of things. 
There is a false sense of urgency that is imposing a workload on a team that is 
providing a free produce and service. The tools for correcting a moment zero 
malware exists in the tool for the operator to use. The real problem is the 
discovery and validation and that is why moment zero solutions will never be 
possible.

There is a finite time required to receive a malware instance, discover it is a 
malware, discover what it applies to, and to create a signature that reasonably 
avoids false positives. I'm convinced that interval exceeds the delay due to 
sync problems by such a margin that the first interval needs as much focus as 
can be committed while the distribution issues are handled at a lower priority.

There are other probabilities - as an example, the probability that a new 
malware is sufficiently in the wild to pose a threat to an important number of 
recipients and which can be very low. Those can be queued for release cutting 
down on the number of low-value updates. And somebody has to decide what is an 
important number.

Evidence of self-replication is recognizable by the rate of increase of 
infestations and is data that can be used in setting priorities. How to collect 
that? How to collect any metrics? So far it is largely buzz generated by 
responders and which is largely anecdotal.

To be honest, many problems would be solved if all outbound mail were scanned in 
real time.

dp

On 10/20/18 8:10 AM, Paul Kosinski wrote:
> Yes, file synchronization is difficult. But we *started out* using the
> provided (i.e., standard) freshclam tool to update our daily.cvd (etc.).
> I only built our current non-standard tool (reading the file header)
> when the Cloudflare mirrors started serving out-of-date file versions
> which caused freshclam to fail and blacklist the mirror (which
> eventually resulted in all mirrors being blacklisted).
>
> This says to me that the old, "standard", DNS TXT approach built in to
> freshclam doesn't play well with Cloudflare (or similar mirrors?).
>
>
>
> On Sat, 20 Oct 2018 06:57:55 -0700
> Dennis Peterson <dennispe at inetnw.com> wrote:
>
>> Caching file systems do validate the requested file against a master
>> file to see if there has been a change. De-dupe caches do the same.
>> It isn't instantaneous but they also don't have to wait for the cache
>> to refresh as they can deliver a pass through request at the same
>> time they're updating the cache. This is more expensive than
>> scheduled sync methods, but those necessarily have a delay. These
>> systems should reject requests for files they don't have but that is
>> difficult if the updated file has the same name as the one it
>> replaces. I know it was always a big deal for the dot com I worked
>> for to update Akamai because of sync problems around the world.
>> Atomic synchronized file updates are pretty much impossible when you
>> have a million page requests/minute.
>>
>> I agree with Joel about using non-standard tools to request
>> signatures and people that do so should have no expectation of
>> consistent high reliability, and support requests should go in the
>> bit bucket. The risk associated with self-service falls on the
>> operator, not the vendor.
>>
>> dp
>>
>> On 10/19/18 2:19 PM, Paul Kosinski wrote:
>>> I'm glad modern multi-core / multi-thread CPU's don't operate this
>>> way.
>>>
>>> Imagine if, when your code on CPU1 tried to access memory location
>>> M, your code got what CPU1 happened to have in its cache, instead
>>> of what CPU2 stored into M a few microseconds ago. Fortunately,
>>> with real CPUs, CPU2 invalidates the other CPUs' caches, and CPU1
>>> takes  the extra time to fetch the new and correct data from memory.
>>>
>>> Thus, what Cloudflare *should* have (if you can't explicitly upload
>>> a file), is a mechanism to tell it that a file is out of date. This
>>> mechanism could operate very quickly. Then, what Cloudflare would
>>> do is either to stall the HTTP response -- I doubt it would have to
>>> stall for long -- or reply with the appropriate HTTP status code
>>> warning the requester that something is amiss. (Codes 503, 504 or
>>> 409 might be applicable.)
>>>
>>>
>>> On Thu, 18 Oct 2018 22:34:03 +0000
>>> "Joel Esler (jesler)" <jesler at cisco.com> wrote:
>>>
>>>> Cloudflare will grab the file from our infrastructure once it's
>>>> been requested.  (Otherwise it wouldn't know it was there, we
>>>> can't push into Cloudflare.). But we have discussed a few ideas
>>>> internally that I think will fix this, let us try a couple things
>>>> and see if it cuts down on this.
>>>>
>>>> On Oct 18, 2018, at 1:55 PM, Eric Tykwinski
>>>> <eric-list at truenet.com<mailto:eric-list at truenet.com>> wrote:
>>>>
>>>> As far as I know you don't upload to cloudflare, it's more of how
>>>> often does cloudflare check to see if the files have changed.
>>>> So you setup a TTL on the check frequency on the cloudflare
>>>> website.
>>>>
>>>> Since updates are new they should just be pulled when you ask from
>>>> the main clam server.
>>>> So you ask for daily-25048.cdiff, and Cloudflare will ask Clam's
>>>> main server for that file and cache it.
>>>>
>>>> So my guess would be same as the TTL on the DNS check:
>>>> current.cvd.clamav.net<http://current.cvd.clamav.net>. 1800
>>>> IN      TXT "0.100.2:58:25048:1539883740:1:63:48006:327"
>>>> I.E. 30 minutes for older files, and new ones are when they come
>>>> in.
>>>>
>>>> Sound about right Joel, Micah?
>>>>
>>>> Sincerely,
>>>>
>>>> Eric Tykwinski
>>>> TrueNet, Inc.
>>>> P: 610-429-8300
>>>>
>>>> -----Original Message-----
>>>> From: clamav-users [mailto:clamav-users-bounces at lists.clamav.net]
>>>> On Behalf Of Paul Kosinski
>>>> Sent: Thursday, October 18, 2018 1:23 PM
>>>> To:
>>>> clamav-users at lists.clamav.net<mailto:clamav-users at lists.clamav.net>
>>>> Subject: Re: [clamav-users] Latest report on update "delays"
>>>>
>>>> How can it take 10, 20 30 or more minutes (and I've seen well over
>>>> an hour at times) to upload the ClamAV database to Cloudflare?
>>>> Does it have to be uploaded separately (and maybe sequentially)
>>>> from Cisco to each Cloudflare mirror? Or is Cloudflare's automatic
>>>> propagation slow?
>>>>
>>>>
>>>> On Thu, 18 Oct 2018 16:07:38 +0000
>>>> "Micah Snyder (micasnyd)"
>>>> <micasnyd at cisco.com<mailto:micasnyd at cisco.com>> wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>> I realize it may look misleading to state that you're up to date
>>>> when a newer database has been announced.  However, if the newer
>>>> database is still being uploaded to the CDN, it is more accurate
>>>> to say that the DNS announcement is premature.
>>>>
>>>> The change to freshclam is an effort to ignore potentially
>>>> premature database version numbers listed via DNS.
>>>>
>>>> Micah Snyder
>>>> ClamAV Development
>>>> Talos
>>>> Cisco Systems, Inc.
> _______________________________________________
> clamav-users mailing list
> clamav-users at lists.clamav.net
> http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml