You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tillrohrmann <gi...@git.apache.org> on 2017/05/10 13:22:03 UTC

[GitHub] flink pull request #3864: [FLINK-6519] Integrate BlobStore in lifecycle mana...

GitHub user tillrohrmann opened a pull request:

    https://github.com/apache/flink/pull/3864

    [FLINK-6519] Integrate BlobStore in lifecycle management of HighAvailabilityServices

    This PR is based on #3512.
    
    The `HighAvailabilityServices` create a single `BlobStoreService` instance which is
    shared by all `BlobServer` and `BlobCache` instances. The `BlobStoreService's` lifecycle
    is exclusively managed by the `HighAvailabilityServices`. This means that the
    `BlobStore's` content is only cleaned up if the `HighAvailabilityServices'` HA data
    is cleaned up. Having this single point of control, makes it easier to decide when
    to discard HA data (e.g. in case of a successful job execution) and when to retain
    the data (e.g. for recovery).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink blobStoreLifecycle

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3864.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3864
    
----
commit 8f4f75fec4af1f941cdd10ad585c49f198a19cea
Author: Nico Kruber <ni...@data-artisans.com>
Date:   2017-01-06T17:42:58Z

    [FLINK-6008] Improve BlobService implementation
    
    [FLINK-6008][docs] minor improvements in the BlobService docs
    
    [FLINK-6008] use Preconditions.checkArgument in BlobClient
    
    [FLINK-6008] refactor BlobCache#getURL() for cleaner code
    
    [FLINK-6008] promote BlobStore#deleteAll(JobID) to the BlobService
    
    [FLINK-6008] extend the BlobService to the NAME_ADDRESSABLE blobs
    
    These blobs are referenced by the job ID and a selected name instead of the
    hash sum of the blob's contents. Some code was already prepared but lacked
    the proper additions in further APIs. This commit adds some.
    
    [FLINK-6008] properly remove NAME_ADDRESSABLE blobs after job/task termination
    
    [FLINK-6008] more unit tests for NAME_ADDRESSABLE and BlobService access
    
    NAME_ADDRESSABLE blobs were not that thouroughly tested before and also the
    access methods that the BlobService implementations provide. This adds tests
    covering both.
    
    [FLINK-6008] do not fail the BlobServer when delete fails
    
    This also enables us to reuse some more code between BlobServerConnection and
    BlobServer.
    
    [FLINK-6008] refactor BlobCache#deleteGlobal() for cleaner code
    
    [FLINK-6008] fix concurrent job directory creation
    
    also add according unit tests
    
    [FLINK-6008] address some of the PR comments by @StephanEwen
    
    [FLINK-6008] some comments about BlobLibraryCacheManager cleanup
    
    [hotfix] minor typos
    
    [FLINK-6008] add retrieval and proper cleanup of name-addressable blobs at the BlobLibraryCacheManager
    
    [FLINK-6008] further cleanup tests for BlobLibraryCacheManager
    
    [FLINK-6008] remove the exposal of the undelying blob service in LibraryCacheManager
    
    This may actually change in future.

commit ae2bea34ec6e00cb43bd6868dcfd577b96006565
Author: Till Rohrmann <tr...@apache.org>
Date:   2017-05-09T08:26:37Z

    [FLINK-6519] Integrate BlobStore in lifecycle management of HighAvailabilityServices
    
    The HighAvailabilityService creates a single BlobStoreService instance which is
    shared by all BlobServer and BlobCache instances. The BlobStoreService's lifecycle
    is exclusively managed by the HighAvailabilityServices. This means that the
    BlobStore's content is only cleaned up if the HighAvailabilityService's HA data
    is cleaned up. Having this single point of control, makes it easier to decide when
    to discard HA data (e.g. in case of a successful job execution) and when to retain
    the data (e.g. for recovery).
    
    Close and cleanup all data of BlobStore in HighAvailabilityServices
    
    Use HighAvailabilityServices to create BlobStore
    
    Introduce BlobStoreService interface to hide close and closeAndCleanupAllData methods

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3864: [FLINK-6519] Integrate BlobStore in lifecycle mana...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/3864


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---