You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "damccorm (via GitHub)" <gi...@apache.org> on 2023/03/01 16:02:45 UTC
[GitHub] [beam] damccorm opened a new issue, #25675: [Bug]:
damccorm opened a new issue, #25675:
URL: https://github.com/apache/beam/issues/25675
### What happened?
https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/ is permared. The failing test is `testWithShardedKeyInGlobalWindow`.
I looked at an example job and it isn't even starting up correctly - https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-03-01_02_18_07-687229064971437925;bottomTab=JOB_LOGS;logsSeverity=ERROR;graphView=0?project=apache-beam-testing&pageState=%28%22dfTime%22:%28%22l%22:%22dfJobMaxTime%22%29%29&e=-13802955&jsmode=o&mods=dataflow_dev - in the Dataflow backend I see Job 2023-03-01_02_18_07-687229064971437925 failed to start with status: generic::invalid_argument: Autosharding is not supported with Unified Worker/Runner v2.
### Issue Priority
Priority: 1 (data loss / total loss of function)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #25675: [Bug]:
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25675:
URL: https://github.com/apache/beam/issues/25675#issuecomment-1450401054
Making this a release blocker until we can verify that it is not an issue with Beam itself. I suspect its an issue with the dataflow runner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #25675: [Feature]: Implement and reenable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to be permared
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25675:
URL: https://github.com/apache/beam/issues/25675#issuecomment-1521808174
Note, this is slightly different than the initial problem since its on runner v1, not runner v2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #25675: [Feature]: Implement and reenable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to be permared
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #25675:
URL: https://github.com/apache/beam/issues/25675#issuecomment-1476850745
I believe @danielarn is working on this from the Dataflow side. Daniel, if you want you could self-assign this so no one else tries to do anything with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #25675: [Feature]: Implement and reenable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to be permared
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #25675:
URL: https://github.com/apache/beam/issues/25675#issuecomment-1521800429
Still permared after test re-enabled: https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/
error message:
```
java.lang.IllegalArgumentException: Runner determined sharding not available in Dataflow for GroupIntoBatches for non-Streaming-Engine jobs. In order to use runner determined sharding, please use --streaming --enable_streaming_engine
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at org.apache.beam.runners.dataflow.DataflowRunner.maybeRecordPCollectionWithAutoSharding(DataflowRunner.java:1644)
at org.apache.beam.runners.dataflow.GroupIntoBatchesOverride$StreamingGroupIntoBatchesWithShardedKey.expand(GroupIntoBatchesOverride.java:335)
at org.apache.beam.runners.dataflow.GroupIntoBatchesOverride$StreamingGroupIntoBatchesWithShardedKey.expand(GroupIntoBatchesOverride.java:314)
at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:576)
```
shall we lift the check argument at the exception path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #25675: [Bug]:
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #25675:
URL: https://github.com/apache/beam/issues/25675#issuecomment-1450433977
Confirmed that this is a known issue with Dataflow runner v2, so I'm sickbaying the test and will cherry pick that to the release branch and remove the release blocker
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #25675: [Feature]: Implement and reenable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to be permared
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #25675:
URL: https://github.com/apache/beam/issues/25675#issuecomment-1521825622
Run Java Dataflow V2 ValidatesRunner Streaming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm closed issue #25675: [Feature]: Implement and reenable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to be permared
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #25675: [Feature]: Implement and reenable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to be permared
URL: https://github.com/apache/beam/issues/25675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn closed issue #25675: [Feature]: Implement and reenable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to be permared
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #25675: [Feature]: Implement and reenable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to be permared
URL: https://github.com/apache/beam/issues/25675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] danielarn commented on issue #25675: [Feature]: Implement and reenable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to be permared
Posted by "danielarn (via GitHub)" <gi...@apache.org>.
danielarn commented on issue #25675:
URL: https://github.com/apache/beam/issues/25675#issuecomment-1476890794
Confirming that I'm actively working on this from the Dataflow side. Will update once autosharding functionality is restored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #25675: [Bug]: Reenable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow: causes dataflow suite to be permared
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #25675:
URL: https://github.com/apache/beam/issues/25675#issuecomment-1476844777
Since this is ongoing feature work in Dataflow, it is P2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org