You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2017/10/17 21:08:22 UTC
[14/14] beam git commit: This closes #3938: [BEAM-2674] Add custom
rehydration; reinstate proto roundtrip for Java DirectRunner
This closes #3938: [BEAM-2674] Add custom rehydration; reinstate proto roundtrip for Java DirectRunner
DirectRunner: Replace use of RawPTransform with NotSerializable.forUrn translators
Clearer getOrDefault style in RehydratedComponents
Add NotSerializable.forUrn to key by URN for non-serializable overrides
Support side inputs in CombineTranslation
Fix typo in UnboundedSource deserialization error message
Reinstate proto round trip in Java DirectRunner
Better error message for RehydratedComponents.getCoder
Add custom rehydration for WriteFiles
Add custom rehydration for ParDo
Add custom rehydration for Combine
Add custom rehydration for TestStream
Add TransformPayloadTranslator.rehydrate to optionally specialize RawPTransform
Add RawPTransform.migrate(SdkComponents) for re-serialization
Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/de7cc05c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/de7cc05c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/de7cc05c
Branch: refs/heads/master
Commit: de7cc05cc67d1aa6331cddc17c2e02ed0efbe37d
Parents: 0f7736d 505021e
Author: Kenneth Knowles <kl...@google.com>
Authored: Tue Oct 17 13:48:57 2017 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Tue Oct 17 13:48:57 2017 -0700
----------------------------------------------------------------------
.../core/construction/CombineTranslation.java | 192 +++++++++++--
.../CreatePCollectionViewTranslation.java | 7 +-
.../core/construction/FlattenTranslator.java | 8 +-
.../construction/GroupByKeyTranslation.java | 13 +-
.../construction/PTransformTranslation.java | 261 +++++++++++++----
.../core/construction/ParDoTranslation.java | 282 ++++++++++++++-----
.../core/construction/PipelineTranslation.java | 110 +-------
.../core/construction/ReadTranslation.java | 43 ++-
.../core/construction/RehydratedComponents.java | 5 +-
.../core/construction/SplittableParDo.java | 8 +
.../construction/TestStreamTranslation.java | 165 +++++++++--
.../TransformPayloadTranslatorRegistrar.java | 2 +
.../construction/WindowIntoTranslation.java | 15 +-
.../construction/WriteFilesTranslation.java | 172 +++++++++--
.../construction/CombineTranslationTest.java | 16 +-
.../construction/TestStreamTranslationTest.java | 4 +-
.../construction/WriteFilesTranslationTest.java | 3 +-
.../core/SplittableParDoViaKeyedWorkItems.java | 7 +
.../beam/runners/direct/DirectGroupByKey.java | 16 +-
.../beam/runners/direct/DirectOptions.java | 8 -
.../beam/runners/direct/DirectRunner.java | 15 +-
.../beam/runners/direct/MultiStepCombine.java | 18 +-
.../direct/ParDoMultiOverrideFactory.java | 9 +-
.../direct/TestStreamEvaluatorFactory.java | 9 +-
.../direct/TransformEvaluatorRegistry.java | 32 +--
.../runners/direct/ViewOverrideFactory.java | 8 +-
runners/flink/pom.xml | 5 -
.../FlinkStreamingTransformTranslators.java | 60 +---
28 files changed, 1017 insertions(+), 476 deletions(-)
----------------------------------------------------------------------