[jira] [Commented] (FLINK-9623) Move zipping logic out of blobservice

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518112#comment-16518112 ] 

ASF GitHub Bot commented on FLINK-9623:
---------------------------------------

GitHub user zentol opened a pull request:

    https://github.com/apache/flink/pull/6187

    FLINK-9623][runtime] Move zipping logic out of blobservice

    ## What is the purpose of the change
    
    This PR moves the zipping logic for local user directory artifacts out of the blobservice.
    Instead, directories are explicitly zipped by the `JobGraphGenerators` when compiling the graph.
    The zipping code was generalized and moved from `FileCache` and `BlobClient` to `FileUtils`.
    
    Additionally, `JobGraphTest` now extends `TestLogger`.
    
    ## Brief change log
    
    * move zip compression/expansion as general utility methods to `FileUtils`
    * modify `[Streaming]JobGraphGenerator` to zip local user directory artifacts before registering them with the graph
    * adjust `FileCacheDirectoryTest` to not manually create a mock zip file
    * extend documentation of `FileCache`
    * extend documentation of `DistributedCache`
    * remove setting of user-artifacts in `TestStreamEnvironment` since it is redundant
    * remove odd verification/normalization code in `Plan#registerCachedFile`, changes are covered by JobGraphGenerators
    
    ## Verifying this change
    
    * zip compression/expansion is covered by new tests in `FileUtils`
    * handling of local user directory artifacts by JobGraphGenerators covered by new tests in `JobGraphGeneratorTest`
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
      - The S3 file system connector: (no)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zentol/flink 9280_alpha

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/6187.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #6187
    
----
commit a8552a5fc85c4a3122eecd7739f5a02065dc93cd
Author: zentol <ch...@...>
Date:   2018-06-13T14:10:16Z

    [hotfix][tests] JobGraphTest extends TestLogger

commit 483ff525a7c047360a5c044fbb3c6bffb08a10c5
Author: zentol <ch...@...>
Date:   2018-06-04T11:50:44Z

    FLINK-9623][runtime] Move zipping logic out of blobservice

----


> Move zipping logic out of blobservice
> -------------------------------------
>
>                 Key: FLINK-9623
>                 URL: https://issues.apache.org/jira/browse/FLINK-9623
>             Project: Flink
>          Issue Type: Improvement
>          Components: Job-Submission
>    Affects Versions: 1.6.0
>            Reporter: Chesnay Schepler
>            Assignee: Chesnay Schepler
>            Priority: Major
>             Fix For: 1.6.0
>
>
> Directories given to the blob-service (primarily a use-case for the distributed cache) are currently silently zipped, and later unzipped by the {{FileCache}}. This tightly coupled the zipping logic in the blob-service to the unzipping logic of the {{FileCache}}. The blob-service neither unzipped the directory if the blob was requested, nor did it provide any means of doing so manually, nor did it inform the user as to whether the requested blob is a zip or not.
> My conclusion in is that the blob-service should not support directories _for now_, and that instead directories for the {{distributed cache}} should be explicitly zipped beforehand, given that this is the only use-case we have at the moment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)