You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "danielarn (via GitHub)" <gi...@apache.org> on 2023/04/10 20:23:19 UTC

[GitHub] [beam] danielarn opened a new pull request, #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

danielarn opened a new pull request, #26207:
URL: https://github.com/apache/beam/pull/26207

   Autosharding support for Java pipelines is now fixed on Dataflow. Un-sickbays GroupIntoBatches tests, fixes #25675.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1502280589

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1521814132

   The change is made for V1 validate runner test task "validatesRunnerStreaming". If it itended to re-enable v2 test then should change here: https://github.com/apache/beam/blob/7a060c5c95f2fbff07bdf455a86705cd6b5e8ee7/runners/google-cloud-dataflow-java/build.gradle#L532


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1507211468

   Run Java Dataflow V2 ValidatesRunner Streaming


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1521799377

   Still permared after test re-enabled: https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/
   
   error message:
   ```
   java.lang.IllegalArgumentException: Runner determined sharding not available in Dataflow for GroupIntoBatches for non-Streaming-Engine jobs. In order to use runner determined sharding, please use --streaming --enable_streaming_engine
   	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
   	at org.apache.beam.runners.dataflow.DataflowRunner.maybeRecordPCollectionWithAutoSharding(DataflowRunner.java:1644)
   	at org.apache.beam.runners.dataflow.GroupIntoBatchesOverride$StreamingGroupIntoBatchesWithShardedKey.expand(GroupIntoBatchesOverride.java:335)
   	at org.apache.beam.runners.dataflow.GroupIntoBatchesOverride$StreamingGroupIntoBatchesWithShardedKey.expand(GroupIntoBatchesOverride.java:314)
   	at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:576)
   ```
   
   shall we lift the check argument at the exception path?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] danielarn commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "danielarn (via GitHub)" <gi...@apache.org>.
danielarn commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1502279618

   R: @damccorm @kennknowles 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm merged pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #26207:
URL: https://github.com/apache/beam/pull/26207


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1521825909

   closed in favor of #26416 which Yi put up at the same time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1521806388

   The failures are non-v2, so this seems different...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1505646994

   Run Java Dataflow V2 ValidatesRunner Streaming


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1521821374

   https://github.com/apache/beam/pull/26415 should fix this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1507211853

   (trying this again since on the first run we ran into lots of quota issues)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1521796688

   This test is still failing after re-enabled: https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming/
   
   Are there additional change need to be made? @danielarn


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on pull request #26207: Re-enable GroupIntoBatchesTest.testWithShardedKeyInGlobalWindow

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #26207:
URL: https://github.com/apache/beam/pull/26207#issuecomment-1521816446

   Yep, that's the issue - thanks @Abacn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org