You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by reuvenlax <gi...@git.apache.org> on 2017/06/14 04:05:22 UTC

[GitHub] beam pull request #3356: [BEAM-92] Allow value-dependent files in FileBasedS...

GitHub user reuvenlax opened a pull request:

    https://github.com/apache/beam/pull/3356

    [BEAM-92] Allow value-dependent files in FileBasedSink

    This is modeled off of the pattern used in BigQueryIO. The user can provide a DynamicDestinations class which can map the input into a user-defined destination type, and the destination into a FilenamePolicy. Some refactoring of FilenamePolicy is done as well to make this more useful - e.g. allowing FilenamePolicy to pick its own base directory instead of having it passed in from the sink (this is marked as @Experimental, so such changes are allowed).
    
    Not yet in this PR: we should allow the sinks to take user-defined types for mapping. We should also provide a convenience method for dynamic output using the DefaultFilenamePolicy (e.g. passing in KVs).
    
    R: @jkff 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/reuvenlax/incubator-beam dynamic_file_based_sink

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3356.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3356
    
----
commit 531a013f6f1e6865566f648d3486d77a7de20369
Author: Reuven Lax <re...@google.com>
Date:   2017-06-10T00:11:32Z

    Add DynamicDestinations support to FileBasedSink

commit 616f9ede5afa7e771f50595bcd290b907744d57f
Author: Reuven Lax <re...@google.com>
Date:   2017-06-10T18:23:32Z

    More fixups.

commit c6855fa130c86c94de74c36976a667de7f8d6219
Author: Reuven Lax <re...@google.com>
Date:   2017-06-12T16:42:26Z

    Fix some tests.

commit f4d4bf5f9238f7e9627363daa14158cdcc68d64b
Author: Reuven Lax <re...@google.com>
Date:   2017-06-13T21:34:08Z

    Remove baseDirectory parameter from FilenamePolicy. The FilenamePolicy can choose it's own base directory.

commit 219a3d649428f673bf35f043990d61000d5fd64e
Author: Reuven Lax <re...@google.com>
Date:   2017-06-14T00:06:01Z

    No longer pass in "extension" to FilenamePolicy. Instead we pass in file metadata class, including a getSuggestedExtension method that is based on the compression type.

commit 1c5ed9e39e891b4418f597d531f442806b4f22b8
Author: Reuven Lax <re...@google.com>
Date:   2017-06-14T00:41:22Z

    Add a withTempDirectory override to TextIO and AvroIO. This way users aren't forced to provide a dummy file prefix, just to specify a temp directory.

commit 14bf19544801ce0791ffd2f95574ba9b5b9d33c6
Author: Reuven Lax <re...@google.com>
Date:   2017-06-14T00:49:34Z

    Fix validation code.

commit 93727530ac83e70570f6da1214fa20c5871b97e6
Author: Reuven Lax <re...@google.com>
Date:   2017-06-14T00:52:08Z

    Fix fix javadoc.

commit da6b0e648fac9d9def26c37aa755b73a5a3f9cef
Author: Reuven Lax <re...@google.com>
Date:   2017-06-14T02:29:00Z

    Fix CheckStyle violations.

commit d2624f729cb7f7dc49f9dda65f6db7744f21e3c6
Author: Reuven Lax <re...@google.com>
Date:   2017-06-14T04:01:26Z

    Fix some failures.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3356: [BEAM-92] Allow value-dependent files in FileBasedS...

Posted by asfgit <gi...@git.apache.org>.

Github user asfgit closed the pull request at:

    https://github.com/apache/beam/pull/3356


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---