You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by mm...@apache.org on 2023/03/15 18:48:10 UTC

[beam] branch master updated (9e044312424 -> 7adcb417f8e)

This is an automated email from the ASF dual-hosted git repository.

mmack pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


    from 9e044312424 Reorganizing the ML content (#25700)
     add 7adcb417f8e [Spark Dataset runner] Implement producer-consumer fusion for ParDos (closes #25761) (#25762)

No new revisions were added by this update.

Summary of changes:
 .../translation/PipelineTranslator.java            |  79 ++++--
 .../translation/TransformTranslator.java           |   7 +
 .../batch/DoFnPartitionIteratorFactory.java        | 131 ++-------
 .../translation/batch/DoFnRunnerFactory.java       | 296 +++++++++++++++++++++
 .../translation/batch/DoFnRunnerWithMetrics.java   |  19 +-
 .../translation/batch/ParDoTranslatorBatch.java    |  64 ++++-
 6 files changed, 451 insertions(+), 145 deletions(-)
 create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnRunnerFactory.java