You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "tuhaihe (via GitHub)" <gi...@apache.org> on 2026/09/16 02:23:30 UTC
[PR] License: add binary license and notice files [cloudberry-backup]
tuhaihe opened a new pull request, #118:
URL: https://github.com/apache/cloudberry-backup/pull/118
The source release bundles no third-party code: there is no vendor/ directory, so LICENSE and NOTICE describe the source tree correctly. The convenience binary packages are a different artifact. They are statically linked Go binaries and therefore physically contain the code of every module in the build graph, plus the Go runtime and standard library and, because CGO is enabled for SQLite support, the SQLite amalgamation.
Until now `make package` copied the source LICENSE and NOTICE verbatim into those packages, where the only third-party component listed is the Greenplum-derived code. That describes neither the 53 Go modules linked into the binaries nor the NOTICE files that eight of them ship, which the Apache License 2.0 requires us to propagate in section 4(d).
Add the -binary variants, following the convention used by Apache Spark, Apache Kafka and apache/cloudberry#1994:
```
LICENSE-binary -> installed as LICENSE
NOTICE-binary -> installed as NOTICE
licenses-binary/ -> installed as licenses/
```
They are generated by scripts/generate-binary-license.sh rather than maintained by hand. The script resolves the modules actually linked into each shipped binary on each released platform, so test-only dependencies such as Ginkgo and Gomega are left out, while platform-gated modules that appear only on Linux are included. A dependency bump that changes the inventory is caught by the new binary-license-check CI job, which runs the script with --check.
The reproduced license texts carry their own upstream copyright notices and must not be given ASF headers, so they are excluded from the RAT scan.
Verified against the actual artifacts: building the packages on Linux and reading the module list back out of the six shipped binaries with `go version -m` matches LICENSE-binary exactly, module for module and version for version.
Backpatch-through: REL_2_STABLE
<!--Thank you for contributing! -->
<!--In case of an existing issue or discussions, please reference it-->
closes: #ISSUE_Number
<!--Remove this section if no corresponding issue.-->
---
## Change logs
> Describe your change clearly, including what problem is being solved or what document is being added or updated.
## Contributor's checklist
Here are some reminders before you submit your pull request:
* Make sure that your Pull Request has a clear title and commit message. You can take the [Git commit template](https://github.com/apache/cloudberry/blob/main/.gitmessage) as a reference.
* Learn the [code contribution](https://cloudberry.apache.org/contribute/code) and [doc contribution](https://cloudberry.apache.org/contribute/doc) guides for better collaboration.
* Make sure that CICD workflow is successful.
* List your communications in the [GitHub Issues](https://github.com/apache/cloudberry-backup/issues) or [Discussions](https://github.com/apache/cloudberry/discussions) (if has or needed).
* Feel free to ask for the [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers) or other people to help review and approve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] License: add binary license and notice files [cloudberry-backup]
Posted by "talmacschen-arch (via GitHub)" <gi...@apache.org>.
talmacschen-arch commented on code in PR #118:
URL: https://github.com/apache/cloudberry-backup/pull/118#discussion_r4414367742
##########
scripts/generate-binary-license.sh:
##########
@@ -0,0 +1,484 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# --------------------------------------------------------------------
+# Generate the compliance files for the binary (convenience) packages.
+#
+# The source release contains none of our Go dependencies: there is no
+# vendor/ directory, so LICENSE and NOTICE correctly describe the source
+# tree on their own. The binaries we ship are statically linked and
+# therefore physically contain the code of every module in the build
+# graph, so the binary packages ship LICENSE-binary and NOTICE-binary
+# instead. They are installed under the plain names LICENSE and NOTICE
+# by the `package` target in the Makefile.
+#
+# The component list is derived from the build graph rather than from
+# go.mod, so that it stays honest across dependency bumps:
+#
+# - only modules linked into a *shipped* binary are listed, so
+# test-only dependencies (ginkgo, gomega, go-sqlmock, ...) are
+# correctly left out;
+# - the union over every released GOOS/GOARCH is taken, because some
+# modules are platform-gated (mdlayher/socket and prometheus/procfs
+# are linked on Linux but not on macOS).
+#
+# Usage:
+# scripts/generate-binary-license.sh # (re)generate in place
+# scripts/generate-binary-license.sh --check # fail if out of date
+# --------------------------------------------------------------------
+
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "${REPO_ROOT}"
+
+# Build tags of the binaries shipped in the convenience packages. Keep in
+# sync with the `package` target in the Makefile.
+BINARY_TAGS=(
+ gpbackup
+ gprestore
+ gpbackup_helper
+ gpbackup_s3_plugin
+ gpbackman
+ gpbackup_exporter
+)
+
+# Platforms built by `make package-all`.
+PLATFORMS=(
+ linux/amd64
+ linux/arm64
+)
+
+CHECK_MODE=0
+case "${1:-}" in
+ --check) CHECK_MODE=1 ;;
+ "") ;;
+ *) echo "usage: $0 [--check]" >&2; exit 2 ;;
+esac
+
+WORK_DIR="$(mktemp -d)"
+trap 'rm -rf "${WORK_DIR}"' EXIT
+
+if (( CHECK_MODE )); then
+ OUT_DIR="${WORK_DIR}/out"
+else
+ OUT_DIR="${REPO_ROOT}"
+fi
+mkdir -p "${OUT_DIR}"
+
+LICENSE_BINARY="${OUT_DIR}/LICENSE-binary"
+NOTICE_BINARY="${OUT_DIR}/NOTICE-binary"
+LICENSES_DIR="${OUT_DIR}/licenses-binary"
+
+# --------------------------------------------------------------------
+# Resolve the modules actually linked into the shipped binaries.
+# --------------------------------------------------------------------
+
+SELF_MODULE="$(go list -m)"
+
+echo "==> Resolving linked modules (${#BINARY_TAGS[@]} binaries x ${#PLATFORMS[@]} platforms)"
+
+MODULES_RAW="${WORK_DIR}/modules.raw"
+: > "${MODULES_RAW}"
+
+for platform in "${PLATFORMS[@]}"; do
+ goos="${platform%/*}"
+ goarch="${platform#*/}"
+ for tag in "${BINARY_TAGS[@]}"; do
+ GOOS="${goos}" GOARCH="${goarch}" CGO_ENABLED=1 \
+ go list -tags "${tag}" -deps \
+ -f '{{if .Module}}{{.Module.Path}} {{.Module.Version}} {{.Module.Dir}}{{end}}' . \
+ >> "${MODULES_RAW}"
+ done
+done
+
+MODULES="${WORK_DIR}/modules"
+grep -v "^${SELF_MODULE} " "${MODULES_RAW}" | sort -u > "${MODULES}"
+
+if [[ ! -s "${MODULES}" ]]; then
+ echo "ERROR: no modules resolved; is the module cache populated? try 'go mod download'" >&2
+ exit 1
+fi
+
+echo "==> $(wc -l < "${MODULES}" | tr -d ' ') linked third-party modules"
+
+# --------------------------------------------------------------------
+# Classify each module's license.
+#
+# A module's primary license is the family whose marker appears first in
+# its license file: several dependencies (klauspost/compress is the
+# clearest case) carry a BSD-licensed body with the Apache and MIT texts
+# appended for vendored parts, and matching on "Apache License" alone
+# would mislabel those.
+# --------------------------------------------------------------------
+
+first_match_line() {
+ # first_match_line <file> <pattern> -> line number, or empty
+ grep -n -i -m1 -- "$2" "$1" 2>/dev/null | cut -d: -f1
+}
+
+classify_license() {
+ local lf="$1" best_family="" best_line="" line
+ local -a families=(
+ "Apache-2.0|Apache License"
+ "MIT|Permission is hereby granted, free of charge"
+ "BSD|Redistributions of source code must retain"
+ "MPL-2.0|Mozilla Public License"
+ "ISC|Permission to use, copy, modify, and/or distribute this software"
+ )
+ local entry family pattern
+ for entry in "${families[@]}"; do
+ family="${entry%%|*}"
+ pattern="${entry#*|}"
+ line="$(first_match_line "${lf}" "${pattern}")"
+ [[ -z "${line}" ]] && continue
+ if [[ -z "${best_line}" || "${line}" -lt "${best_line}" ]]; then
+ best_line="${line}"
+ best_family="${family}"
+ fi
+ done
+ if [[ "${best_family}" == "BSD" ]]; then
+ if grep -q -i -e "Neither the name" -e "name of the author" "${lf}"; then
+ best_family="BSD-3-Clause"
+ else
+ best_family="BSD-2-Clause"
+ fi
+ fi
+ echo "${best_family:-UNKNOWN}"
+}
+
+has_multiple_licenses() {
+ # true when one license file carries more than one license family
+ local lf="$1" count=0 entry pattern
+ local -a patterns=(
+ "Apache License"
+ "Permission is hereby granted, free of charge"
+ "Redistributions of source code must retain"
+ "Mozilla Public License"
+ )
+ for pattern in "${patterns[@]}"; do
+ if grep -q -i -- "${pattern}" "${lf}" 2>/dev/null; then
+ count=$(( count + 1 ))
+ fi
+ done
+ (( count > 1 ))
+}
+
+rm -rf "${LICENSES_DIR}"
+mkdir -p "${LICENSES_DIR}"
+
+INVENTORY="${WORK_DIR}/inventory" # family \t module \t version \t relpath \t flags
+NOTICES="${WORK_DIR}/notices" # module \t version \t notice file
+: > "${INVENTORY}"
+: > "${NOTICES}"
+
+while IFS=$'\t' read -r mod ver dir; do
+ [[ -z "${mod}" ]] && continue
+
+ # Kept in a file rather than an array: this script has to run under the
+ # bash 3.2 that ships with macOS, which has no mapfile.
+ license_list="${WORK_DIR}/license_files"
+ find "${dir}" -maxdepth 1 -type f \
+ \( -iname 'LICENSE' -o -iname 'LICENSE.*' \
+ -o -iname 'LICENCE' -o -iname 'LICENCE.*' \
+ -o -iname 'COPYING' -o -iname 'COPYING.*' \) 2>/dev/null | sort > "${license_list}"
+
+ license_count="$(wc -l < "${license_list}" | tr -d ' ')"
+ if [[ "${license_count}" -eq 0 ]]; then
+ echo "ERROR: no license file found for ${mod} ${ver} (${dir})" >&2
+ echo " inspect the module and record its license by hand" >&2
+ exit 1
+ fi
+
+ dest_dir="${LICENSES_DIR}/${mod}"
+ mkdir -p "${dest_dir}"
+
+ primary=""
+ extra_note=""
+ while IFS= read -r lf; do
+ [[ -z "${lf}" ]] && continue
+ base="$(basename "${lf}")"
+ cp "${lf}" "${dest_dir}/${base}"
+ chmod 0644 "${dest_dir}/${base}"
+ if [[ -z "${primary}" ]]; then
+ primary="$(classify_license "${lf}")"
+ if has_multiple_licenses "${lf}"; then
+ extra_note="contains additional license terms for bundled code"
+ fi
+ fi
+ done < "${license_list}"
+
+ if [[ "${license_count}" -gt 1 ]]; then
+ extra_note="multiple license files; see licenses/${mod}/"
+ fi
+
+ if [[ "${primary}" == "UNKNOWN" ]]; then
+ echo "ERROR: could not classify the license of ${mod} ${ver}" >&2
+ echo " inspect ${license_files[0]} and extend classify_license()" >&2
Review Comment:
`${license_files[0]}` is an undefined variable. The script uses `license_list` (a file path) and the loop variable `lf` throughout — `license_files` as an array is never defined.
Since the script starts with `set -euo pipefail` (including `-u`), as soon as a dependency's license can't be classified and this `UNKNOWN` branch is hit, `${license_files[0]}` aborts with `license_files[0]: unbound variable` instead of printing the intended `inspect <file>` hint — a misleading error exactly when a clear diagnostic is most needed.
Suggest pointing at the actual license directory, e.g.:
```sh
echo " inspect the license files under ${dest_dir} and extend classify_license()" >&2
```
(or `${license_list}`.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] License: add binary license and notice files [cloudberry-backup]
Posted by "talmacschen-arch (via GitHub)" <gi...@apache.org>.
talmacschen-arch commented on code in PR #118:
URL: https://github.com/apache/cloudberry-backup/pull/118#discussion_r4414367742
##########
scripts/generate-binary-license.sh:
##########
@@ -0,0 +1,484 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# --------------------------------------------------------------------
+# Generate the compliance files for the binary (convenience) packages.
+#
+# The source release contains none of our Go dependencies: there is no
+# vendor/ directory, so LICENSE and NOTICE correctly describe the source
+# tree on their own. The binaries we ship are statically linked and
+# therefore physically contain the code of every module in the build
+# graph, so the binary packages ship LICENSE-binary and NOTICE-binary
+# instead. They are installed under the plain names LICENSE and NOTICE
+# by the `package` target in the Makefile.
+#
+# The component list is derived from the build graph rather than from
+# go.mod, so that it stays honest across dependency bumps:
+#
+# - only modules linked into a *shipped* binary are listed, so
+# test-only dependencies (ginkgo, gomega, go-sqlmock, ...) are
+# correctly left out;
+# - the union over every released GOOS/GOARCH is taken, because some
+# modules are platform-gated (mdlayher/socket and prometheus/procfs
+# are linked on Linux but not on macOS).
+#
+# Usage:
+# scripts/generate-binary-license.sh # (re)generate in place
+# scripts/generate-binary-license.sh --check # fail if out of date
+# --------------------------------------------------------------------
+
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "${REPO_ROOT}"
+
+# Build tags of the binaries shipped in the convenience packages. Keep in
+# sync with the `package` target in the Makefile.
+BINARY_TAGS=(
+ gpbackup
+ gprestore
+ gpbackup_helper
+ gpbackup_s3_plugin
+ gpbackman
+ gpbackup_exporter
+)
+
+# Platforms built by `make package-all`.
+PLATFORMS=(
+ linux/amd64
+ linux/arm64
+)
+
+CHECK_MODE=0
+case "${1:-}" in
+ --check) CHECK_MODE=1 ;;
+ "") ;;
+ *) echo "usage: $0 [--check]" >&2; exit 2 ;;
+esac
+
+WORK_DIR="$(mktemp -d)"
+trap 'rm -rf "${WORK_DIR}"' EXIT
+
+if (( CHECK_MODE )); then
+ OUT_DIR="${WORK_DIR}/out"
+else
+ OUT_DIR="${REPO_ROOT}"
+fi
+mkdir -p "${OUT_DIR}"
+
+LICENSE_BINARY="${OUT_DIR}/LICENSE-binary"
+NOTICE_BINARY="${OUT_DIR}/NOTICE-binary"
+LICENSES_DIR="${OUT_DIR}/licenses-binary"
+
+# --------------------------------------------------------------------
+# Resolve the modules actually linked into the shipped binaries.
+# --------------------------------------------------------------------
+
+SELF_MODULE="$(go list -m)"
+
+echo "==> Resolving linked modules (${#BINARY_TAGS[@]} binaries x ${#PLATFORMS[@]} platforms)"
+
+MODULES_RAW="${WORK_DIR}/modules.raw"
+: > "${MODULES_RAW}"
+
+for platform in "${PLATFORMS[@]}"; do
+ goos="${platform%/*}"
+ goarch="${platform#*/}"
+ for tag in "${BINARY_TAGS[@]}"; do
+ GOOS="${goos}" GOARCH="${goarch}" CGO_ENABLED=1 \
+ go list -tags "${tag}" -deps \
+ -f '{{if .Module}}{{.Module.Path}} {{.Module.Version}} {{.Module.Dir}}{{end}}' . \
+ >> "${MODULES_RAW}"
+ done
+done
+
+MODULES="${WORK_DIR}/modules"
+grep -v "^${SELF_MODULE} " "${MODULES_RAW}" | sort -u > "${MODULES}"
+
+if [[ ! -s "${MODULES}" ]]; then
+ echo "ERROR: no modules resolved; is the module cache populated? try 'go mod download'" >&2
+ exit 1
+fi
+
+echo "==> $(wc -l < "${MODULES}" | tr -d ' ') linked third-party modules"
+
+# --------------------------------------------------------------------
+# Classify each module's license.
+#
+# A module's primary license is the family whose marker appears first in
+# its license file: several dependencies (klauspost/compress is the
+# clearest case) carry a BSD-licensed body with the Apache and MIT texts
+# appended for vendored parts, and matching on "Apache License" alone
+# would mislabel those.
+# --------------------------------------------------------------------
+
+first_match_line() {
+ # first_match_line <file> <pattern> -> line number, or empty
+ grep -n -i -m1 -- "$2" "$1" 2>/dev/null | cut -d: -f1
+}
+
+classify_license() {
+ local lf="$1" best_family="" best_line="" line
+ local -a families=(
+ "Apache-2.0|Apache License"
+ "MIT|Permission is hereby granted, free of charge"
+ "BSD|Redistributions of source code must retain"
+ "MPL-2.0|Mozilla Public License"
+ "ISC|Permission to use, copy, modify, and/or distribute this software"
+ )
+ local entry family pattern
+ for entry in "${families[@]}"; do
+ family="${entry%%|*}"
+ pattern="${entry#*|}"
+ line="$(first_match_line "${lf}" "${pattern}")"
+ [[ -z "${line}" ]] && continue
+ if [[ -z "${best_line}" || "${line}" -lt "${best_line}" ]]; then
+ best_line="${line}"
+ best_family="${family}"
+ fi
+ done
+ if [[ "${best_family}" == "BSD" ]]; then
+ if grep -q -i -e "Neither the name" -e "name of the author" "${lf}"; then
+ best_family="BSD-3-Clause"
+ else
+ best_family="BSD-2-Clause"
+ fi
+ fi
+ echo "${best_family:-UNKNOWN}"
+}
+
+has_multiple_licenses() {
+ # true when one license file carries more than one license family
+ local lf="$1" count=0 entry pattern
+ local -a patterns=(
+ "Apache License"
+ "Permission is hereby granted, free of charge"
+ "Redistributions of source code must retain"
+ "Mozilla Public License"
+ )
+ for pattern in "${patterns[@]}"; do
+ if grep -q -i -- "${pattern}" "${lf}" 2>/dev/null; then
+ count=$(( count + 1 ))
+ fi
+ done
+ (( count > 1 ))
+}
+
+rm -rf "${LICENSES_DIR}"
+mkdir -p "${LICENSES_DIR}"
+
+INVENTORY="${WORK_DIR}/inventory" # family \t module \t version \t relpath \t flags
+NOTICES="${WORK_DIR}/notices" # module \t version \t notice file
+: > "${INVENTORY}"
+: > "${NOTICES}"
+
+while IFS=$'\t' read -r mod ver dir; do
+ [[ -z "${mod}" ]] && continue
+
+ # Kept in a file rather than an array: this script has to run under the
+ # bash 3.2 that ships with macOS, which has no mapfile.
+ license_list="${WORK_DIR}/license_files"
+ find "${dir}" -maxdepth 1 -type f \
+ \( -iname 'LICENSE' -o -iname 'LICENSE.*' \
+ -o -iname 'LICENCE' -o -iname 'LICENCE.*' \
+ -o -iname 'COPYING' -o -iname 'COPYING.*' \) 2>/dev/null | sort > "${license_list}"
+
+ license_count="$(wc -l < "${license_list}" | tr -d ' ')"
+ if [[ "${license_count}" -eq 0 ]]; then
+ echo "ERROR: no license file found for ${mod} ${ver} (${dir})" >&2
+ echo " inspect the module and record its license by hand" >&2
+ exit 1
+ fi
+
+ dest_dir="${LICENSES_DIR}/${mod}"
+ mkdir -p "${dest_dir}"
+
+ primary=""
+ extra_note=""
+ while IFS= read -r lf; do
+ [[ -z "${lf}" ]] && continue
+ base="$(basename "${lf}")"
+ cp "${lf}" "${dest_dir}/${base}"
+ chmod 0644 "${dest_dir}/${base}"
+ if [[ -z "${primary}" ]]; then
+ primary="$(classify_license "${lf}")"
+ if has_multiple_licenses "${lf}"; then
+ extra_note="contains additional license terms for bundled code"
+ fi
+ fi
+ done < "${license_list}"
+
+ if [[ "${license_count}" -gt 1 ]]; then
+ extra_note="multiple license files; see licenses/${mod}/"
+ fi
+
+ if [[ "${primary}" == "UNKNOWN" ]]; then
+ echo "ERROR: could not classify the license of ${mod} ${ver}" >&2
+ echo " inspect ${license_files[0]} and extend classify_license()" >&2
Review Comment:
`${license_files[0]}` 是未定义变量。脚本全程用的是 `license_list`（文件路径）和循环变量 `lf`，从未定义过 `license_files` 这个数组。
脚本开头是 `set -euo pipefail`（含 `-u`），所以一旦某个依赖的 license 无法被 `classify_license` 识别而走到这个 `UNKNOWN` 分支，`${license_files[0]}` 会先抛 `license_files[0]: unbound variable` 崩溃，而不是打印本意的 `inspect <file>` 诊断——恰恰在最需要清晰提示的时刻给出误导性错误。
建议指向实际的 license 目录，例如：
```sh
echo " inspect the license files under ${dest_dir} and extend classify_license()" >&2
```
（或 `${license_list}`。）
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org