[jira] [Assigned] (HDDS-17275) Make release tarballs reproducible across build environments

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng reassigned HDDS-17275:
---------------------------------

    Assignee: Siyao Meng

> Make release tarballs reproducible across build environments
> ------------------------------------------------------------
>
>                 Key: HDDS-17275
>                 URL: https://issues.apache.org/jira/browse/HDDS-17275
>             Project: Apache Ozone
>          Issue Type: Sub-task
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>
> h3. Problem
> The {{repro}} check runs {{artifact:compare}} in the same job, on the same runner, path, JDK, user, locale and umask as the reference build. It covers attached Maven artifacts, but not the release tarballs ({{ozone-X.tar.gz}} is produced by {{dist-tar-stitching}}, {{ozone-X-src.tar.gz}} has {{attach=false}}), and it cannot detect differences caused by the build environment. Verifying a release independently (for example with Apache Trusted Releases, see HADOOP-19976) needs the tarballs to be identical when built elsewhere.
> h3. Observation
> Two runs of {{mvn clean package -DskipTests -DskipDocs -Pdist -Psrc}} on the same machine, the second with {{TZ=Asia/Tokyo LC_ALL=de_DE.UTF-8}} and {{umask 002}}:
> * all jars (including javadoc, sources and shaded jars), {{bom.json}}, {{bom.xml}} and the source tarball were identical
> * the binary tarball had identical file contents and entry order, but different archive metadata: entry mtimes, permissions (following umask), owner and group names of the builder, and the mtime and file name in the gzip header
> h3. Gaps
> # *Binary tarball metadata.* {{dist-tar-stitching}} runs plain {{tar}} and {{gzip}}. Entry order also depends on the directory listing order of the filesystem (sorted on APFS, not on ext4).
> # *Unpinned inputs.* The {{hadoop-native-lib}} profile downloads libhadoop from the {{master}} branch of {{ozone-thirdparty}} without a checksum. The RocksDB source archive for {{hdds-rocks-native}} is downloaded without a checksum.
> # *Optional content depending on the environment.* Docs are included only if Hugo is available, and {{--enableGitInfo}} is used only in a git checkout, so building from the source tarball gives different docs than building from git. libhadoop is included only if it was downloaded. The Hugo docs are being removed by HDDS-15923 ([PR #10829|https://github.com/apache/ozone/pull/10829]), which resolves the docs part.
> # *Build environment not pinned or recorded.* The release guide uses the floating {{rockylinux:9}} image and unpinned JDK, cmake and gcc packages. No {{.buildinfo}} is produced, so a verifier cannot know which toolchain to use.
> # *Native library build path.* {{libozone_rocksdb_tools.so}} is built without {{-ffile-prefix-map}}, so it may embed the build directory. Not verified.
> # *rpm and deb packages* (HDDS-13439) may embed build host and build time. Not verified; only relevant if they become release artifacts.
> h3. Proposed changes
> * Create the binary tarball with sorted entries, mtime from {{project.build.outputTimestamp}}, numeric owner and group 0, normalized permissions, and {{gzip -n}}.
> * Pin downloaded inputs to a tag or commit and verify them with {{sha256}}.
> * Make tarball content independent of optional downloads such as libhadoop (include it consistently or fail).
> * Generate {{.buildinfo}} ({{artifact:buildinfo}}) and document a pinned build environment for releases.
> * Extend the {{repro}} check (or add a separate one) to compare both tarballs from an independent build with a different path, user, TZ, locale and umask.
> These can be split into separate subtasks if preferred. Nondeterminism within a single build is tracked separately in HDDS-16309 and HDDS-12862.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org