You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/12/02 18:03:57 UTC
[GitHub] [beam] runningcode opened a new pull request #13464: [BEAM-11383] Add runtime normalization for sdk.properties.
runningcode opened a new pull request #13464:
URL: https://github.com/apache/beam/pull/13464
This normalizes the sdk.properties file which has a build timestamp in it as well as the version and sdk version. For the purposes of the runtime classpath, these can be ignored so that tasks are not rerun if only the build timestamp is changed.
This should result in significantly higher cache hits as well as more up-to-date checks.
This also preserves timestamps for certain tasks for the same reason.
BEAM-11383
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam
.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #13464: [BEAM-11383] Add runtime normalization for file timestamps
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #13464:
URL: https://github.com/apache/beam/pull/13464#issuecomment-739028022
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] runningcode commented on a change in pull request #13464: [BEAM-11383] Add runtime normalization for file timestamps
Posted by GitBox <gi...@apache.org>.
runningcode commented on a change in pull request #13464:
URL: https://github.com/apache/beam/pull/13464#discussion_r535961637
##########
File path: build.gradle
##########
@@ -375,8 +375,16 @@ if (project.hasProperty('compileAndRunTestsWithJava11')) {
project.sqlPreCommit.dependsOn ':sdks:java:testing:test-utils:verifyJavaVersion'
} else {
allprojects {
- tasks.withType(Test) {
+ tasks.withType(Test).configureEach {
exclude '**/JvmVerification.class'
}
}
}
+
+allprojects {
Review comment:
Added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #13464: [BEAM-11383] Add runtime normalization for file timestamps
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #13464:
URL: https://github.com/apache/beam/pull/13464
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #13464: [BEAM-11383] Add runtime normalization for file timestamps
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #13464:
URL: https://github.com/apache/beam/pull/13464#discussion_r535734866
##########
File path: build.gradle
##########
@@ -375,8 +375,16 @@ if (project.hasProperty('compileAndRunTestsWithJava11')) {
project.sqlPreCommit.dependsOn ':sdks:java:testing:test-utils:verifyJavaVersion'
} else {
allprojects {
- tasks.withType(Test) {
+ tasks.withType(Test).configureEach {
exclude '**/JvmVerification.class'
}
}
}
+
+allprojects {
Review comment:
nit:
```suggestion
# Ignore sdk.properties so it doesn't spoil the build cache unnecessarily (see https://docs.gradle.org/6.6/userguide/more_about_tasks.html#sec:configure_input_normalization)
allprojects {
```
(or something)
##########
File path: sdks/java/core/build.gradle
##########
@@ -66,6 +66,10 @@ test {
}
}
+tasks.withType(Jar).configureEach {
+ preserveFileTimestamps(false)
+}
+
Review comment:
Could/should we do this in applyJavaNature and/or applyPortabilityNature to get the benefit for all built jars?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #13464: [BEAM-11383] Add runtime normalization for file timestamps
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #13464:
URL: https://github.com/apache/beam/pull/13464#discussion_r536388649
##########
File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##########
@@ -770,6 +770,10 @@ class BeamModulePlugin implements Plugin<Project> {
+ (defaultLintSuppressions + configuration.disableLintWarnings).collect { "-Xlint:-${it}" })
}
+ project.tasks.withType(Jar).configureEach {
+ preserveFileTimestamps(false)
+ }
+
Review comment:
Ah yes I guess applyJavaNature is called in applyPortabilityNature already so this caputres both :+1:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] runningcode commented on pull request #13464: [BEAM-11383] Add runtime normalization for file timestamps
Posted by GitBox <gi...@apache.org>.
runningcode commented on pull request #13464:
URL: https://github.com/apache/beam/pull/13464#issuecomment-737535320
R: @TheNeuralBit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] runningcode commented on a change in pull request #13464: [BEAM-11383] Add runtime normalization for file timestamps
Posted by GitBox <gi...@apache.org>.
runningcode commented on a change in pull request #13464:
URL: https://github.com/apache/beam/pull/13464#discussion_r535962206
##########
File path: sdks/java/core/build.gradle
##########
@@ -66,6 +66,10 @@ test {
}
}
+tasks.withType(Jar).configureEach {
+ preserveFileTimestamps(false)
+}
+
Review comment:
Good point. I added this to `applyJavaNature` since `applyPortabilityNature` also calls `applyJavaNature`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org