[jira] [Commented] (MESOS-2073) Fetcher cache file verification, updating and invalidation

["Bernd Mathiske (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14653531#comment-14653531 ] 

Bernd Mathiske commented on MESOS-2073:
---------------------------------------

In phase 1, the server should enact dynamic content updates as follows.

1. Remove the checksum file. This will cause the fetcher to not cache content until we are in a stable situation again.
2. Swap in the new content file.
3. Put a fresh checksum file in place.

This way there is no race between content and checksum preventing cache refreshing when it must be indicated. However, the saved checksum may not actually correspond to the downloaded content. It may be older. This is tolerable, because 
    - in any case newer content than previously in the cache did get downloaded and
    - if the checksum is indeed out of sync, the next request will see either no checksum or a newer checksum 
        and hence refresh the cache again.
This means that in no case an update is missed. Worst case, the same content gets downloaded twice, once with the old checksum and then again with the correct one. This is also tolerable.

In case multiple updates happen while holding on to an outdated checksum, this also works out OK, as long as every new checksum is different from all others. 

If the latter cannot be guaranteed, for instance because the checksum calculations in use are idempotent and old content may reappear, then we have a problem. Two approaches may help:
A) Look at "Last-Modified" headers and such to still establish different identities of equal-looking checksums. 
B) Download the checksum before AND after downloading the content. Only if it matches, proceed. Otherwise start over.

Approach A strongly hints that phase 1 would be obsolete if we simply honored HTTP cache directives. But this is a protocol-specific approach. I suggest that in the long run we implement both. But we start with phase 1, as this leads us to phase 2, which I reckon is what we typically really want.

Note (again) that in phase 1 the checksum is allowed to say nothing about the integrity of the downloaded content. For example, simple serial numbers can be used.


> Fetcher cache file verification, updating and invalidation
> ----------------------------------------------------------
>
>                 Key: MESOS-2073
>                 URL: https://issues.apache.org/jira/browse/MESOS-2073
>             Project: Mesos
>          Issue Type: Improvement
>          Components: fetcher, slave
>            Reporter: Bernd Mathiske
>            Assignee: Bernd Mathiske
>            Priority: Minor
>              Labels: mesosphere
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> The other tickets in the fetcher cache epic do not necessitate a check sum (e.g. MD5, SHA*) for files cached by the fetcher. Whereas such a check sum could be used to verify whether the file arrived without unintended alterations, it can first and foremost be employed to detect and trigger updates. 
> Scenario: If a UIR is requested for fetching and the indicated download has the same check sum as the cached file, then the cache file will be used and the download forgone. If the check sum is different, then fetching proceeds and the cached file gets replaced. 
> This capability will be indicated by an additional field in the URI protobuf. Details TBD, i.e. to be discussed in comments below.
> In addition to the above, even if the check sum is the same, we can support voluntary cache file invalidation: a fresh download can be requested, or the caching behavior can be revoked entirely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)