You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/15 23:00:17 UTC
[GitHub] [beam] AlexZMLyu opened a new pull request, #21903: Add streaming test for Write API sink
AlexZMLyu opened a new pull request, #21903:
URL: https://github.com/apache/beam/pull/21903
Add streaming test for BigQuery Write API
@chamikaramj
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1232176710
Run PostCommit_Java_DataflowV1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1157028871
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1244990694
Spotless test passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1212615129
Run PostCommit_Java_DataflowV1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1234642552
Run PostCommit_Java_DataflowV1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1157028872
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1194362434
Run PostCommit_Java_DataflowV2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1162419988
Talked to @pabloem offline. The TestStream class that I used here as the streaming source is not compatible with Dataflow runner. I will edit the code using the **GenerateSequence** and overriding the **expand** method to serve as the streaming source instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1212613874
Seems like Storage Write API tests are still failing for Runner v2: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV2_PR/111/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryIOStorageWriteIT/testBigQueryStorageWrite30MProto/
So we can just get the Runner v1 version in (this PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
robertwb commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1185846676
TestStream should work fine on Runner v2, and is preferable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1194508444
There was no failed test in https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV2_PR/112/testReport/
Why the Dataflow V2 Java Post Commit Tests failed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1208574371
Run PostCommit_Java_DataflowV2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1212743435
Seems like the new tests failed the post-commit test suite: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV1_PR/124/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1245792758
Looks like all test passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1245794309
Thanks. Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1191829846
This test is disabled for Runner v2 currently: https://github.com/apache/beam/blob/67e6726ffeb47d2ada0122369fa230833ce0f026/runners/google-cloud-dataflow-java/build.gradle#L642
Trying to re-enable here: https://github.com/apache/beam/pull/21814
(you can try it on this PR similarly)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1157028881
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1215847696
Then I still need to change the streaming source from TestStream to something else, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1191821791
Robert, thanks for letting me know.
Do I need to change my code to utilize the Runner V2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1245664698
Run Java_GCP_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1157029742
@chamikaramj Hi Cham, please kindly review the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1217373744
Yeah, I think it's probably ok to use GenerateSequence (a bounded source) for a simple integration test that use the streaming Runner v1. Separately we should check why this still failed for Runner v2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yirutang commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
yirutang commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1157068475
R: @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1212613960
Retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1244752996
Run Spotless PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1245665109
Seems like there's still one failing test suite. Probably unrelated. Retrying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1162090456
Running postcommit that exercises these tests: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV1/1765/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj merged pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj merged PR #21903:
URL: https://github.com/apache/beam/pull/21903
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1208573709
Do I need to wait approval of [PR #21814 ](https://github.com/apache/beam/pull/21814) to run Runner v2 here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1212640340
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1157028876
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1212615667
Run PostCommit_Java_Dataflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1157028878
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1162033788
@AlexZMLyu please run `./gradlew :sdks:java:io:google-cloud-platform:spotlessApply` to format the code and fix spotless issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1162032877
Run Java PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AlexZMLyu commented on pull request #21903: Add streaming test for Write API sink
Posted by GitBox <gi...@apache.org>.
AlexZMLyu commented on PR #21903:
URL: https://github.com/apache/beam/pull/21903#issuecomment-1162020089
R: @pabloem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org