You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/09/26 00:12:05 UTC
[GitHub] [beam] tysonjh opened a new pull request #12947: [Beam-10903] Fix Java nightly snapshot build.
tysonjh opened a new pull request #12947:
URL: https://github.com/apache/beam/pull/12947
After upgrading to Gradle 6.6.1 (commit
1db383c58c214ec2e10edec72e7fc427c74ebb1c) the snapshot builds started
failing. This is because the maven plugin now publishes Maven 3
metadata, where previously it published Maven 2 metadata. From the
userguide [1]:
The Maven Plugin used to publish the highly outdated Maven 2 metadata format. This has
been changed and it will now publish Maven 3 metadata, just like the Maven Publish Plugin.
With the removal of Maven 2 support, the methods that configure unique snapshot behavior
have also been removed. Maven 3 only supports unique snapshots, so we decided to remove
them.
[1]: https://docs.gradle.org/current/userguide/userguide.pdf
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #12947: [Beam-10903] Fix Java nightly snapshot build.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #12947:
URL: https://github.com/apache/beam/pull/12947#issuecomment-700344217
> Makes sense. How did you track it down, out of curiosity? Browsing https://ci-beam.apache.org/job/beam_Release_NightlySnapshot/935/ and thereabouts didn't yield such a clear result while https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/ is healthy (I don't know what exactly these jobs actually do in detail, and particularly the latter name makes no sense to me)
I tracked it down by throwing a debugger up on intellij and walking through the build.gradle file until I found the exception location in the Maven plugin (org/gradle/api/publish/maven/internal/publisher/AbstractMavenPublisher.java:75) where the Array OOB exception occurred. Then I put a break point in the build.gradle file (neat!) and eventually traced it to the snapshot metadata missing, opened up the maven-metadata.xml file and saw it was missing snapshot 'versions'. From there I compared it with the published metadata, got confused for a while, then found the user guide (https://docs.gradle.org/current/userguide/userguide.pdf) that talks about changes with the maven publish plugin. Eventually I found that there were some breaking changes mentioned in the guide, saw the Beam commit history that correlated with the breakage, and *POW*. After that I tried a few things, using the user guide examples with gratuitous searching of the word 'snapshot', and ended up here. There is also
some information in the online user guide about components (e.g. the 'from components.java' bit) here: https://docs.gradle.org/current/userguide/publishing_customization.html.
Regarding the jobs --
Those are two different jobs. The 'PostRelease_NightlySnapshot' runs validation tests against the snapshot that was *released by* the 'Release_NightlySnapshot' artifacts. So the 'PostRelease_NightlySnapshot' job has been running against an old snapshot from Sept. 9th (the last successful snapshot published).
For example see the lastupdated date in the examples metadata (20200928070145): https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-maven-archetypes-examples/2.26.0-SNAPSHOT/maven-metadata.xml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #12947: [Beam-10903] Fix Java nightly snapshot build.
Posted by GitBox <gi...@apache.org>.
kennknowles commented on pull request #12947:
URL: https://github.com/apache/beam/pull/12947#issuecomment-699304359
Makes sense. How did you track it down, out of curiosity? Browsing https://ci-beam.apache.org/job/beam_Release_NightlySnapshot/935/ and thereabouts didn't yield such a clear result while https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/ is healthy (I don't know what exactly these jobs actually do in detail, and particularly the latter name makes no sense to me)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #12947: [Beam-10903] Fix Java nightly snapshot build.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #12947:
URL: https://github.com/apache/beam/pull/12947#issuecomment-700349425
> Gradle bit LGTM though I don't have deep understanding. Thanks for the comments. The commit history isn't clean.
Fixed. The gradle bit isn't very well documented, from what I read, it is a plugin that can be used to ensure proper publishing of the artifacts. Which in this case was used to fix the metadata-xml file that is generated for the snapshots. I was unable to separate fixing the snapshot metadata from generating the jar file. Without generating the jar file, the publishing fails, since the maven java publishing plugin validates that it is produced.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles merged pull request #12947: [Beam-10903] Fix Java nightly snapshot build.
Posted by GitBox <gi...@apache.org>.
kennknowles merged pull request #12947:
URL: https://github.com/apache/beam/pull/12947
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tysonjh commented on pull request #12947: [Beam-10903] Fix Java nightly snapshot build.
Posted by GitBox <gi...@apache.org>.
tysonjh commented on pull request #12947:
URL: https://github.com/apache/beam/pull/12947#issuecomment-699237958
R: @kennknowles
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org