You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/11/02 00:21:00 UTC
[jira] [Work logged] (BEAM-13067) GroupIntoBatches should be
considered a key-preserving operation by Dataflow
[ https://issues.apache.org/jira/browse/BEAM-13067?focusedWorklogId=672919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672919 ]
ASF GitHub Bot logged work on BEAM-13067:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 02/Nov/21 00:20
Start Date: 02/Nov/21 00:20
Worklog Time Spent: 10m
Work Description: reuvenlax commented on a change in pull request #15731:
URL: https://github.com/apache/beam/pull/15731#discussion_r740639484
##########
File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
##########
@@ -716,13 +716,13 @@ private void addOutput(String name, PValue value, Coder<?> valueCoder) {
translator.registerOutputName(value, name);
// If the output requires runner determined sharding, also append necessary input properties.
- if (value instanceof PCollection
- && translator.runner.doesPCollectionRequireAutoSharding((PCollection<?>) value)) {
- addInput(PropertyNames.ALLOWS_SHARDABLE_STATE, "true");
- // Currently we only allow auto-sharding to be enabled through the GroupIntoBatches
- // transform. So we also add the following property which GroupIntoBatchesDoFn has, to allow
- // the backend to perform graph optimization.
- addInput(PropertyNames.PRESERVES_KEYS, "true");
+ if (value instanceof PCollection) {
+ if (translator.runner.doesPCollectionRequireAutoSharding((PCollection<?>) value)) {
+ addInput(PropertyNames.ALLOWS_SHARDABLE_STATE, "true");
+ }
+ if (translator.runner.doesPCollectionPreserveKeys((PCollection<?>) value)) {
+ addInput(PropertyNames.PRESERVES_KEYS, "true");
Review comment:
Would like to generalize and also make this a model feature. There are some other things worth adding as well, such as PARTITIONED_BY_KEY (right now the model doesn't really know if a PCollection<KV> is already partitioned by key. runners can infer it by looking for a GroupByKey followed by only key-preserving operations. however what is a source produces items already partitioned by key?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 672919)
Time Spent: 1h 50m (was: 1h 40m)
> GroupIntoBatches should be considered a key-preserving operation by Dataflow
> ----------------------------------------------------------------------------
>
> Key: BEAM-13067
> URL: https://issues.apache.org/jira/browse/BEAM-13067
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Affects Versions: 2.33.0
> Reporter: Reuven Lax
> Priority: P2
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Dataflow today only marks GroupIntoBatches as key preserving if auto sharding is enabled. It should mark it as key preserving regardless.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)