You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ie...@apache.org on 2019/07/30 09:18:46 UTC
[beam] 01/01: Merge pull request #8946: [BEAM-7574] fix Combine
performance for SparkRunner
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit fb83467970b333d6bdc5af4f7a6f0e88fee63232
Merge: 83bb3dd 1691935
Author: Ismaël Mejía <ie...@gmail.com>
AuthorDate: Tue Jul 30 11:18:08 2019 +0200
Merge pull request #8946: [BEAM-7574] fix Combine performance for SparkRunner
runners/spark/build.gradle | 1 +
.../spark/coders/SparkRunnerKryoRegistrator.java | 2 -
.../spark/translation/GroupCombineFunctions.java | 110 +--
.../GroupNonMergingWindowsFunctions.java | 202 ++---
.../spark/translation/SparkAbstractCombineFn.java | 124 ----
.../SparkBatchPortablePipelineTranslator.java | 4 +-
.../runners/spark/translation/SparkCombineFn.java | 825 +++++++++++++++++++++
.../spark/translation/SparkGlobalCombineFn.java | 261 -------
.../spark/translation/SparkKeyedCombineFn.java | 278 -------
.../spark/translation/TransformTranslator.java | 49 +-
.../spark/translation/TranslationUtils.java | 12 +-
.../streaming/StreamingTransformTranslator.java | 6 +-
.../beam/runners/spark/io/NumShardsTest.java | 2 +-
.../GroupNonMergingWindowsFunctionsTest.java | 124 +++-
.../spark/translation/SparkCombineFnTest.java | 298 ++++++++
.../org/apache/beam/sdk/options/ValueProvider.java | 1 +
.../org/apache/beam/sdk/transforms/Combine.java | 5 +
.../beam/sdk/transforms/windowing/Sessions.java | 6 +
.../org/apache/beam/sdk/util/WindowedValue.java | 55 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 20 +
20 files changed, 1497 insertions(+), 888 deletions(-)