You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "adoroszlai (via GitHub)" <gi...@apache.org> on 2024/06/23 17:20:16 UTC
[PR] HDDS-11057. Enable reproducible builds [ozone]
adoroszlai opened a new pull request, #6856:
URL: https://github.com/apache/ozone/pull/6856
## What changes were proposed in this pull request?
- Enable [reproducible builds](https://maven.apache.org/guides/mini/guide-reproducible-builds.html) by setting the `project.build.outputTimestamp` property.
- Remove build user and date/time, which make builds non-reproducible.
- Add new check `repro` to verify jars created by `build` check.
(This is only the first step in making all Ozone build artifacts reproducible.)
https://issues.apache.org/jira/browse/HDDS-11057
## How was this patch tested?
https://github.com/adoroszlai/ozone/actions/runs/9634357696
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11057. Enable reproducible builds [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6856:
URL: https://github.com/apache/ozone/pull/6856#issuecomment-2423079397
Thanks @jojochuang, @kerneltime, @sadanand48 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11057. Enable reproducible builds [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6856:
URL: https://github.com/apache/ozone/pull/6856#issuecomment-2408966860
Thanks @jojochuang for the questions.
> Have we verified it does create reproducible builds?
Yes. The PR adds a new CI check (`repro`) to rebuild Ozone and compare artifacts against the reference created by the `build` check. Comparison is performed by Maven Artifact Plugin's `compare` goal, which reports differences like:
```
[ERROR] size mismatch hdds-common-2.0.0-SNAPSHOT.jar: investigate with diffoscope hadoop-hdds/common/target/reference/org.apache.ozone/hdds-common-2.0.0-SNAPSHOT.jar hadoop-hdds/common/target/hdds-common-2.0.0-SNAPSHOT.jar
[ERROR] Reproducible Build output summary: 6 files ok, 1 different
[ERROR] see diff hadoop-hdds/common/target/reference/hdds-common-2.0.0-SNAPSHOT.buildinfo hadoop-hdds/common/target/hdds-common-2.0.0-SNAPSHOT.buildinfo
[ERROR] see also https://maven.apache.org/guides/mini/guide-reproducible-builds.html
```
When looking this up to answer your question, I just found that `repro` check was passing despite the error. This is now fixed, check is [failing properly](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951) when mismatch is found.
The check also has a follow-up step to display the root cause of the difference, using `diffoscope`. [Example](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:12):
```
--- /home/runner/.m2/repository/org/apache/ozone/hdds-common/2.0.0-SNAPSHOT/hdds-common-2.0.0-SNAPSHOT.jar
+++ hadoop-hdds/common/target/hdds-common-2.0.0-SNAPSHOT.jar
├── zipinfo /dev/stdin
│ @@ -1,8 +1,8 @@
│ -Zip file size: 705376 bytes, number of entries: 430
│ +Zip file size: 705373 bytes, number of entries: 430
│ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 META-INF/
│ -rw-r--r-- 2.0 unx 81 b- defN 23-Jan-01 00:00 META-INF/MANIFEST.MF
│ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 com/
│ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 com/google/
│ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 com/google/protobuf/
│ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 org/
│ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 org/apache/
│ @@ -68,15 +68,15 @@
│ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 META-INF/maven/
│ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 META-INF/maven/org.apache.ozone/
│ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 META-INF/maven/org.apache.ozone/hdds-common/
│ -rw-r--r-- 2.0 unx [12](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:13)660 b- defN 23-Jan-01 00:00 META-INF/LICENSE.txt
│ -rw-r--r-- 2.0 unx 1493 b- defN 23-Jan-01 00:00 META-INF/NOTICE.txt
│ -rw-r--r-- 2.0 unx 628 b- defN 23-Jan-01 00:00 com/google/protobuf/Proto2Utils.class
│ -rw-r--r-- 2.0 unx 3847 b- defN 23-Jan-01 00:00 hdds-common.classpath
│ --rw-r--r-- 2.0 unx 1089 b- defN 23-Jan-01 00:00 hdds-version-info.properties
│ +-rw-r--r-- 2.0 unx 1078 b- defN 23-Jan-01 00:00 hdds-version-info.properties
│ -rw-r--r-- 2.0 unx 3176 b- defN 23-Jan-01 00:00 network-topology-default.xml
│ -rw-r--r-- 2.0 unx 1689 b- defN 23-Jan-01 00:00 network-topology-default.yaml
│ -rw-r--r-- 2.0 unx 3380 b- defN 23-Jan-01 00:00 network-topology-nodegroup.xml
│ -rw-r--r-- 2.0 unx 210 b- defN 23-Jan-01 00:00 org/apache/hadoop/hdds/ComponentVersion.class
│ -rw-r--r-- 2.0 unx 2076 b- defN 23-Jan-01 00:00 org/apache/hadoop/hdds/DFSConfigKeysLegacy.class
│ -rw-r--r-- 2.0 unx 3698 b- defN 23-Jan-01 00:00 org/apache/hadoop/hdds/DatanodeVersion.class
│ -rw-r--r-- 2.0 unx 12469 b- defN 23-Jan-01 00:00 org/apache/hadoop/hdds/HddsConfigKeys.class
│ @@ -425,8 +425,8 @@
│ -rw-r--r-- 2.0 unx 9599 b- defN 23-Jan-01 00:00 org/apache/hadoop/ozone/util/ShutdownHookManager.class
│ -rw-r--r-- 2.0 unx 1414 b- defN 23-Jan-01 00:00 org/apache/hadoop/ozone/util/StringWithByteString.class
│ -rw-r--r-- 2.0 unx 1423 b- defN 23-Jan-01 00:00 org/apache/hadoop/ozone/util/UUIDUtil.class
│ -rw-r--r-- 2.0 unx 11999 b- defN 23-Jan-01 00:00 ozone-default-generated.xml
│ -rw-r--r-- 2.0 unx 163523 b- defN 23-Jan-01 00:00 ozone-default.xml
│ -rw-r--r-- 2.0 unx 1[13](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:14)99 b- defN 23-Jan-01 00:00 META-INF/maven/org.apache.ozone/hdds-common/pom.xml
│ -rw-r--r-- 2.0 unx 71 b- defN 23-Jan-01 00:00 META-INF/maven/org.apache.ozone/hdds-common/pom.properties
│ -430 files, 1670413 bytes uncompressed, 626710 bytes compressed: 62.5%
│ +430 files, 1670402 bytes uncompressed, 626707 bytes compressed: 62.5%
├── hdds-version-info.properties
│ @@ -[14](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:15),14 +14,14 @@
│ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
│ # See the License for the specific language governing permissions and
│ # limitations under the License.
│ #
│
│ version=2.0.0-SNAPSHOT
│ revision=c0b94741e9e[17](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:18)3206855148885afdde62fe8a1e1
│ -branch=(HEAD detached at c0b9474)
│ +branch=HDDS-11057-test
│ url=https://github.com/adoroszlai/ozone
│ srcChecksum=2e[19](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:20)8fe64d737c693739f29336d2341
│ hadoopProtoc2Version=2.5.0
│ hadoopProtoc3Version=3.7.1
│ grpcProtocVersion=3.19.6
│ compilePlatform=linux-x86_64
```
> Supposedly two builds made at different time results in the same tarball checksum?
Currently we only check at the jar level. Tarballs are not yet reproducible due to timestamps at least. I'll check separately if they can be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11057. Enable reproducible builds [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6856:
URL: https://github.com/apache/ozone/pull/6856#discussion_r1805240909
##########
hadoop-ozone/common/src/main/resources/ozone-version-info.properties:
##########
@@ -19,9 +19,6 @@
version=${declared.ozone.version}
release=${ozone.release}
revision=${version-info.scm.commit}
-branch=${version-info.scm.branch}
-user=${user.name}
-date=${version-info.build.time}
url=${version-info.scm.uri}
Review Comment:
You are right, we may need to remove `url` eventually. For the current check it's OK, rebuild happens in the same repo.
Commit ID is worth including even if it causes jar to be different after e.g. commit+revert. I think that carries some information.
https://reproducible-builds.org/docs/version-information/
> is there a CI run where the repro check reported the jars causing discrepancy
Yes, I linked to such a run in my previous comment: https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951
Here `hdds-common-2.0.0-SNAPSHOT.jar` is not the same due to difference in `hdds-version-info.properties`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11057. Enable reproducible builds [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #6856:
URL: https://github.com/apache/ozone/pull/6856#issuecomment-2406164313
Have we verified it does create reproducible builds? Supposedly two builds made at different time results in the same tarball checksum?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11057. Enable reproducible builds [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #6856:
URL: https://github.com/apache/ozone/pull/6856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11057. Enable reproducible builds [ozone]
Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on code in PR #6856:
URL: https://github.com/apache/ozone/pull/6856#discussion_r1805151301
##########
hadoop-ozone/common/src/main/resources/ozone-version-info.properties:
##########
@@ -19,9 +19,6 @@
version=${declared.ozone.version}
release=${ozone.release}
revision=${version-info.scm.commit}
-branch=${version-info.scm.branch}
-user=${user.name}
-date=${version-info.build.time}
url=${version-info.scm.uri}
Review Comment:
sparing commit id and url means the build should happen from official apache repo itself i.e we cannot compare across forks or with same src code but with different commit chain?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org