You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/10 20:55:41 UTC
[GitHub] [beam] andoni-guzman opened a new pull request, #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
andoni-guzman opened a new pull request, #21808:
URL: https://github.com/apache/beam/pull/21808
Sharding IO tests Amazon web services and Amazon web services 2 from Java PostCommit task
Looking to reduce JavaPostCommit task some test were split in different ones. This PR is for Amazon web services and Amazon web services 2 tests.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andoni-guzman commented on a diff in pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
andoni-guzman commented on code in PR #21808:
URL: https://github.com/apache/beam/pull/21808#discussion_r898491178
##########
build.gradle.kts:
##########
@@ -197,8 +197,7 @@ tasks.register("javaPostCommit") {
dependsOn(":sdks:java:extensions:google-cloud-platform-core:postCommit")
dependsOn(":sdks:java:extensions:zetasketch:postCommit")
dependsOn(":sdks:java:io:google-cloud-platform:postCommit")
- dependsOn(":sdks:java:io:amazon-web-services:integrationTest")
- dependsOn(":sdks:java:io:amazon-web-services2:integrationTest")
+ dependsOn(":sdks:java:io:kinesis:integrationTest")
Review Comment:
I think is related to a merge conflict with other branch that is working in the tests. Let me talk about it with the team and do the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andoni-guzman commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
andoni-guzman commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1156944222
Hi @kennknowles
I delete the trigger for direct runner. Please let me know if you want to separate the task and in how way you want it or if it's okay in this way.
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
kennknowles commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1157656063
I suppose the possible reason not to would be if iterating in the PR causes a lot of not-useful builds that create large cost on a real cloud. But I think it will be barely measurable difference
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on a diff in pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
kennknowles commented on code in PR #21808:
URL: https://github.com/apache/beam/pull/21808#discussion_r897075802
##########
.test-infra/jenkins/job_PreCommit_Java_AWS_IO_Direct.groovy:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PrecommitJobBuilder
+
+PrecommitJobBuilder builder = new PrecommitJobBuilder(
+ scope: this,
+ nameBase: 'Java_Amazon-Web-Services_IO_Direct',
+ gradleTask: ':sdks:java:io:amazon-web-services:integrationTest',
+ gradleSwitches: [
+ '-PdisableSpotlessCheck=true'
+ ], // spotless checked in separate pre-commit
+ triggerPathPatterns: [
+ '^sdks/java/io/amazon-web-services/.*$',
+ '^runners/direct-java/.*$',
Review Comment:
Remove this trigger path too.
##########
.test-infra/jenkins/job_PreCommit_Java_AWS2_IO_Direct.groovy:
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PrecommitJobBuilder
+
+PrecommitJobBuilder builder = new PrecommitJobBuilder(
+ scope: this,
+ nameBase: 'Java_Amazon-Web-Services2_IO_Direct',
+ gradleTask: 'sdks:java:io:amazon-web-services2:integrationTest',
+ gradleSwitches: [
+ '-PdisableSpotlessCheck=true'
+ ], // spotless checked in separate pre-commit
+ triggerPathPatterns: [
+ '^sdks/java/io/amazon-web-services2/.*$',
+ '^runners/direct-java/.*$',
Review Comment:
We do not need to run these tests when the direct runner is modified, since these tests are not for checking the correctness of the direct runner. So should remove this trigger path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
kennknowles commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1157654591
Andoni & I talked about it, and I _think_ since it only triggers when the IO itself is being modified, and it is pretty small, that it could be OK to be presubmit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
kennknowles commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1157015130
I suggest doing a `git rebase origin/master` and then force pushing to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1152749095
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1152748507
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1157307494
@andoni-guzman wondering, should this be using the PostcommitJobBuilder, or is it intentional to have it as a precommit job from now on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on a diff in pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
kennknowles commented on code in PR #21808:
URL: https://github.com/apache/beam/pull/21808#discussion_r898489155
##########
build.gradle.kts:
##########
@@ -197,8 +197,7 @@ tasks.register("javaPostCommit") {
dependsOn(":sdks:java:extensions:google-cloud-platform-core:postCommit")
dependsOn(":sdks:java:extensions:zetasketch:postCommit")
dependsOn(":sdks:java:io:google-cloud-platform:postCommit")
- dependsOn(":sdks:java:io:amazon-web-services:integrationTest")
- dependsOn(":sdks:java:io:amazon-web-services2:integrationTest")
+ dependsOn(":sdks:java:io:kinesis:integrationTest")
Review Comment:
Why is this added?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1152748505
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
kennknowles commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1155464436
LGTM other than that. I do think it will be good to separate the tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1152748504
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1153895648
R: @mosche
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] andoni-guzman commented on pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
andoni-guzman commented on PR #21808:
URL: https://github.com/apache/beam/pull/21808#issuecomment-1165880931
Hi @kennknowles,
I think it's ready for review, let me know if something it's missing c:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles merged pull request #21808: Sharding IO tests(amazon web services and amazon web services 2) from java post commit task
Posted by GitBox <gi...@apache.org>.
kennknowles merged PR #21808:
URL: https://github.com/apache/beam/pull/21808
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org