You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "jrmccluskey (via GitHub)" <gi...@apache.org> on 2023/05/31 19:31:42 UTC

[GitHub] [beam] jrmccluskey opened a new pull request, #26954: Remove Beam Dependency Report code

jrmccluskey opened a new pull request, #26954:
URL: https://github.com/apache/beam/pull/26954

   Deletes the Beam Dependency Report Jenkins job and associated code. 
   
   Fixes #26896
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] jrmccluskey commented on pull request #26954: Remove Beam Dependency Report code

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26954:
URL: https://github.com/apache/beam/pull/26954#issuecomment-1572457619

   R: @damccorm 
   
   Is this everything that needs to be done to remove the job? As far as the python code I believe all of the references to it are in the script


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] jrmccluskey commented on pull request #26954: Remove Beam Dependency Report code

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26954:
URL: https://github.com/apache/beam/pull/26954#issuecomment-1578941174

   Run Python_Integration PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] jrmccluskey commented on a diff in pull request #26954: Remove Beam Dependency Report code

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #26954:
URL: https://github.com/apache/beam/pull/26954#discussion_r1213515730


##########
build.gradle.kts:
##########
@@ -574,11 +574,6 @@ tasks.register("javaExamplesDataflowPrecommit") {
   dependsOn(":runners:google-cloud-dataflow-java:examples-streaming:preCommit")
 }
 
-tasks.register("runBeamDependencyCheck") {
-  dependsOn(":dependencyUpdates")

Review Comment:
   Good catch on the unnecessary extra dependency



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] jrmccluskey commented on a diff in pull request #26954: Remove Beam Dependency Report code

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #26954:
URL: https://github.com/apache/beam/pull/26954#discussion_r1213496504


##########
.test-infra/jenkins/job_Dependency_Check.groovy:
##########
@@ -1,73 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-
-job('beam_Dependency_Check') {
-  description('Runs Beam dependency check.')
-
-  // Set common parameters.
-  commonJobProperties.setTopLevelMainJobProperties(
-      delegate, 'master', 100, true, 'beam', false)
-
-  // Allows triggering this build against pull requests.
-  commonJobProperties.enablePhraseTriggeringFromPullRequest(
-      delegate,
-      'Beam Dependency Check',
-      'Run Dependency Check',
-      false
-      )
-
-  commonJobProperties.setAutoJob(
-      delegate,
-      '@weekly')
-
-  steps {
-    gradle {
-      rootBuildScriptDir(commonJobProperties.checkoutDir)
-      tasks('runBeamDependencyCheck')

Review Comment:
   I think I knocked them all out by checking `dependecyCheck`, `dependency-check`, and `dependencyUpdate` references along with the two direct references to the shell script itself. Fortunately touching things within the `sdks/` folder triggers more of the automated testing so there will at least be an early signal of if this really breaks something 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #26954: Remove Beam Dependency Report code

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #26954:
URL: https://github.com/apache/beam/pull/26954#issuecomment-1572578649

   ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/26954?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#26954](https://app.codecov.io/gh/apache/beam/pull/26954?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (b74486f) into [master](https://app.codecov.io/gh/apache/beam/commit/d8f0e1786f1c0d6799783a3cb4a145f2a5d8d94d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d8f0e17) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #26954      +/-   ##
   ==========================================
   - Coverage   71.27%   71.27%   -0.01%     
   ==========================================
     Files         766      766              
     Lines      103470   103480      +10     
   ==========================================
   + Hits        73747    73754       +7     
   - Misses      28184    28187       +3     
     Partials     1539     1539              
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `80.85% <ø> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   [see 7 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/26954/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on a diff in pull request #26954: Remove Beam Dependency Report code

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #26954:
URL: https://github.com/apache/beam/pull/26954#discussion_r1213485807


##########
.test-infra/jenkins/job_Dependency_Check.groovy:
##########
@@ -1,73 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-
-job('beam_Dependency_Check') {
-  description('Runs Beam dependency check.')
-
-  // Set common parameters.
-  commonJobProperties.setTopLevelMainJobProperties(
-      delegate, 'master', 100, true, 'beam', false)
-
-  // Allows triggering this build against pull requests.
-  commonJobProperties.enablePhraseTriggeringFromPullRequest(
-      delegate,
-      'Beam Dependency Check',
-      'Run Dependency Check',
-      false
-      )
-
-  commonJobProperties.setAutoJob(
-      delegate,
-      '@weekly')
-
-  steps {
-    gradle {
-      rootBuildScriptDir(commonJobProperties.checkoutDir)
-      tasks('runBeamDependencyCheck')

Review Comment:
   Sorry, missed that initially



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #26954: Remove Beam Dependency Report code

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26954:
URL: https://github.com/apache/beam/pull/26954#issuecomment-1572460263

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] jrmccluskey merged pull request #26954: Remove Beam Dependency Report code

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey merged PR #26954:
URL: https://github.com/apache/beam/pull/26954


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on a diff in pull request #26954: Remove Beam Dependency Report code

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #26954:
URL: https://github.com/apache/beam/pull/26954#discussion_r1213485508


##########
.test-infra/jenkins/job_Dependency_Check.groovy:
##########
@@ -1,73 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import CommonJobProperties as commonJobProperties
-
-job('beam_Dependency_Check') {
-  description('Runs Beam dependency check.')
-
-  // Set common parameters.
-  commonJobProperties.setTopLevelMainJobProperties(
-      delegate, 'master', 100, true, 'beam', false)
-
-  // Allows triggering this build against pull requests.
-  commonJobProperties.enablePhraseTriggeringFromPullRequest(
-      delegate,
-      'Beam Dependency Check',
-      'Run Dependency Check',
-      false
-      )
-
-  commonJobProperties.setAutoJob(
-      delegate,
-      '@weekly')
-
-  steps {
-    gradle {
-      rootBuildScriptDir(commonJobProperties.checkoutDir)
-      tasks('runBeamDependencyCheck')

Review Comment:
   Actually, I think we need to rip out this piece too.
   
   This has a gradle task https://github.com/apache/beam/blob/4f07cda050cd144a95a686c4138307e9920f313d/build.gradle.kts#L583 which then calls into the python sdk dependency check code - https://github.com/apache/beam/blob/4f07cda050cd144a95a686c4138307e9920f313d/sdks/python/scripts/run_dependency_check.sh#L22
   
   There's some more references that will need cleaned up around this, https://github.com/search?q=repo%3Aapache%2Fbeam%20dependencyUpdates&type=code is a good starting point



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] jrmccluskey commented on a diff in pull request #26954: Remove Beam Dependency Report code

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #26954:
URL: https://github.com/apache/beam/pull/26954#discussion_r1213516310


##########
sdks/python/tox.ini:
##########
@@ -244,16 +244,6 @@ commands =
   yapf --version
   time yapf --diff --parallel --recursive apache_beam
 
-[testenv:py3-dependency-check]
-# TODO(https://github.com/apache/beam/issues/20337): botocore, a part of [aws], wants docutils<0.16, but Sphinx
-# pulls in the latest docutils. Uncomment this line once botocore does not
-# conflict with Sphinx:
-# extras = docs,test,gcp,aws,interactive,interactive_test
-extras = test,gcp,aws,dataframe,interactive,interactive_test
-passenv = WORKSPACE
-commands =
-  time {toxinidir}/scripts/run_dependency_check.sh

Review Comment:
   Didn't realize there was another script in the Python SDK, I had already cleared out the one in the .test-infra directory



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] damccorm commented on a diff in pull request #26954: Remove Beam Dependency Report code

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #26954:
URL: https://github.com/apache/beam/pull/26954#discussion_r1213507660


##########
sdks/python/tox.ini:
##########
@@ -244,16 +244,6 @@ commands =
   yapf --version
   time yapf --diff --parallel --recursive apache_beam
 
-[testenv:py3-dependency-check]
-# TODO(https://github.com/apache/beam/issues/20337): botocore, a part of [aws], wants docutils<0.16, but Sphinx
-# pulls in the latest docutils. Uncomment this line once botocore does not
-# conflict with Sphinx:
-# extras = docs,test,gcp,aws,interactive,interactive_test
-extras = test,gcp,aws,dataframe,interactive,interactive_test
-passenv = WORKSPACE
-commands =
-  time {toxinidir}/scripts/run_dependency_check.sh

Review Comment:
   We should be able to get rid of `run_dependency_check.sh`, right? https://github.com/apache/beam/blob/8ec667c8a0b746849eed6b0199e41070c49a110b/sdks/python/scripts/run_dependency_check.sh#L4



##########
build.gradle.kts:
##########
@@ -574,11 +574,6 @@ tasks.register("javaExamplesDataflowPrecommit") {
   dependsOn(":runners:google-cloud-dataflow-java:examples-streaming:preCommit")
 }
 
-tasks.register("runBeamDependencyCheck") {
-  dependsOn(":dependencyUpdates")

Review Comment:
   Looks like this comes from https://github.com/ben-manes/gradle-versions-plugin so I think we can kill https://github.com/apache/beam/blob/8ec667c8a0b746849eed6b0199e41070c49a110b/build.gradle.kts#L25



##########
build.gradle.kts:
##########
@@ -574,11 +574,6 @@ tasks.register("javaExamplesDataflowPrecommit") {
   dependsOn(":runners:google-cloud-dataflow-java:examples-streaming:preCommit")
 }
 
-tasks.register("runBeamDependencyCheck") {
-  dependsOn(":dependencyUpdates")

Review Comment:
   and https://github.com/apache/beam/blob/8ec667c8a0b746849eed6b0199e41070c49a110b/ownership/JAVA_DEPENDENCY_OWNERS.yaml#L148



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] jrmccluskey commented on pull request #26954: Remove Beam Dependency Report code

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26954:
URL: https://github.com/apache/beam/pull/26954#issuecomment-1572519005

   Done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] jrmccluskey commented on pull request #26954: Remove Beam Dependency Report code

Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #26954:
URL: https://github.com/apache/beam/pull/26954#issuecomment-1579090079

   R: @Abacn 
   
   Re-routing this to you while Danny is out


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org