You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ie...@apache.org on 2019/07/30 09:18:45 UTC
[beam] branch master updated (83bb3dd -> fb83467)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 83bb3dd [BEAM-7700] Java transform catalog (#9129)
add 3ee9da9 [BEAM-7574] Use Map-based window combining
add 6c1d115 [BEAM-7574] use Merging and NonMerging accumulating strategies
add 945a606 [BEAM-7574] unify Combine.perKey and Combine.globally
add 7be5dff [BEAM-7574] Explode windows to key when needed
add 1691935 [BEAM-7574] Optimize GC performance, finalize implementation
new fb83467 Merge pull request #8946: [BEAM-7574] fix Combine performance for SparkRunner
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
runners/spark/build.gradle | 1 +
.../spark/coders/SparkRunnerKryoRegistrator.java | 2 -
.../spark/translation/GroupCombineFunctions.java | 110 +--
.../GroupNonMergingWindowsFunctions.java | 202 ++---
.../spark/translation/SparkAbstractCombineFn.java | 124 ----
.../SparkBatchPortablePipelineTranslator.java | 4 +-
.../runners/spark/translation/SparkCombineFn.java | 825 +++++++++++++++++++++
.../spark/translation/SparkGlobalCombineFn.java | 261 -------
.../spark/translation/SparkKeyedCombineFn.java | 278 -------
.../spark/translation/TransformTranslator.java | 49 +-
.../spark/translation/TranslationUtils.java | 12 +-
.../streaming/StreamingTransformTranslator.java | 6 +-
.../beam/runners/spark/io/NumShardsTest.java | 2 +-
.../GroupNonMergingWindowsFunctionsTest.java | 124 +++-
.../spark/translation/SparkCombineFnTest.java | 298 ++++++++
.../org/apache/beam/sdk/options/ValueProvider.java | 1 +
.../org/apache/beam/sdk/transforms/Combine.java | 5 +
.../beam/sdk/transforms/windowing/Sessions.java | 6 +
.../org/apache/beam/sdk/util/WindowedValue.java | 55 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 20 +
20 files changed, 1497 insertions(+), 888 deletions(-)
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkAbstractCombineFn.java
create mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkCombineFn.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkKeyedCombineFn.java
create mode 100644 runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SparkCombineFnTest.java
[beam] 01/01: Merge pull request #8946: [BEAM-7574] fix Combine
performance for SparkRunner
Posted by ie...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit fb83467970b333d6bdc5af4f7a6f0e88fee63232
Merge: 83bb3dd 1691935
Author: Ismaël Mejía <ie...@gmail.com>
AuthorDate: Tue Jul 30 11:18:08 2019 +0200
Merge pull request #8946: [BEAM-7574] fix Combine performance for SparkRunner
runners/spark/build.gradle | 1 +
.../spark/coders/SparkRunnerKryoRegistrator.java | 2 -
.../spark/translation/GroupCombineFunctions.java | 110 +--
.../GroupNonMergingWindowsFunctions.java | 202 ++---
.../spark/translation/SparkAbstractCombineFn.java | 124 ----
.../SparkBatchPortablePipelineTranslator.java | 4 +-
.../runners/spark/translation/SparkCombineFn.java | 825 +++++++++++++++++++++
.../spark/translation/SparkGlobalCombineFn.java | 261 -------
.../spark/translation/SparkKeyedCombineFn.java | 278 -------
.../spark/translation/TransformTranslator.java | 49 +-
.../spark/translation/TranslationUtils.java | 12 +-
.../streaming/StreamingTransformTranslator.java | 6 +-
.../beam/runners/spark/io/NumShardsTest.java | 2 +-
.../GroupNonMergingWindowsFunctionsTest.java | 124 +++-
.../spark/translation/SparkCombineFnTest.java | 298 ++++++++
.../org/apache/beam/sdk/options/ValueProvider.java | 1 +
.../org/apache/beam/sdk/transforms/Combine.java | 5 +
.../beam/sdk/transforms/windowing/Sessions.java | 6 +
.../org/apache/beam/sdk/util/WindowedValue.java | 55 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 20 +
20 files changed, 1497 insertions(+), 888 deletions(-)