You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "raulcd (via GitHub)" <gi...@apache.org> on 2025/09/19 09:33:15 UTC
[PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
raulcd opened a new pull request, #47600:
URL: https://github.com/apache/arrow/pull/47600
### TBD
### Rationale for this change
### What changes are included in this PR?
### Are these changes tested?
Some via CI on fork and some manual testing.
### Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2372329238
##########
dev/release/utils-watch-gh-workflow.sh:
##########
@@ -30,17 +30,30 @@ TAG=$1
WORKFLOW=$2
: "${REPOSITORY:=${GITHUB_REPOSITORY:-apache/arrow}}"
+get_run_id() {
+ gh run list \
+ --branch "${TAG}" \
+ --jq '.[].databaseId' \
+ --json databaseId \
+ --limit 1 \
+ --repo "${REPOSITORY}" \
+ --workflow "${WORKFLOW}"
+}
+
+download_artifacts() {
+ RUN_ID=$(get_run_id)
+ echo "Downloading artitfacts for workflow with ID: ${RUN_ID}"
+ gh run download \
+ ${RUN_ID} \
+ --repo "${REPOSITORY}" \
+ --dir "$1"
+}
Review Comment:
We could rename the file to `utils-gh-workflow.sh` but if we use release for the artifacts this won't be necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #47600:
URL: https://github.com/apache/arrow/pull/47600#issuecomment-3371300694
Testing RC tag on my fork:
* https://github.com/kou/arrow/releases/tag/apache-arrow-22.0.0-rc4
* https://github.com/kou/arrow/actions/runs/18273060925/job/52039071031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2409797346
##########
.github/workflows/package_linux.yml:
##########
@@ -0,0 +1,313 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Package Linux
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ tags:
+ - "apache-arrow-*-rc*"
+ pull_request:
+ paths:
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ # Upload to GitHub Release
+ contents: write
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ parent-workflow: package_linux
+
+ package:
+ name: ${{ matrix.id }}
+ runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
+ needs: check-labels
+ if: >-
+ needs.check-labels.outputs.force == 'true' ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: Package: Linux')
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ id:
+ - almalinux-8-amd64
+ - almalinux-8-arm64
+ - almalinux-9-amd64
+ - almalinux-9-arm64
+ - almalinux-10-amd64
+ - almalinux-10-arm64
+ - amazon-linux-2023-amd64
+ - amazon-linux-2023-arm64
+ - centos-9-stream-amd64
+ - centos-9-stream-arm64
+ - centos-7-amd64
+ - debian-bookworm-amd64
+ - debian-bookworm-arm64
+ - debian-trixie-amd64
+ - debian-trixie-arm64
+ - debian-forky-amd64
+ - debian-forky-arm64
+ - ubuntu-jammy-amd64
+ - ubuntu-jammy-arm64
+ - ubuntu-noble-amd64
+ - ubuntu-noble-arm64
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Prepare environment variables
+ env:
+ ID: ${{ matrix.id }}
+ run: |
+ set -x
+ case "${ID}" in
+ centos-*)
+ # Example: centos-9-stream-amd64 -> centos
+ distribution="${ID%%-*}"
+ ;;
+ *)
+ # Example: almalinux-8-amd64 -> almalinux
+ # Example: amazon-linux-2023-amd64 -> amazon-linux
+ distribution="${ID%-*-*}"
+ ;;
+ esac
+ echo "DISTRIBUTION=${distribution}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> amd64
+ architecture="${ID##*-}"
+ echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> almalinux-8
+ target="${ID%-*}"
+ case "${target}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: almalinux-8 -> almalinux-8-aarch64
+ target="${target}-aarch64"
+ fi
+ echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ *)
+ echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: ubuntu-noble -> ubuntu-noble-arm64
+ target="${target}-arm64"
+ fi
+ echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ esac
+ echo "TARGET=${target}" >> "${GITHUB_ENV}"
+
+ if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+ # Example: apache-arrow-21.0.0-rc0 -> 21.0.0-rc0
+ version="${GITHUB_REF_NAME#apache-arrow-}"
+ echo "ARROW_VERSION=${version}" >> "${GITHUB_ENV}"
+ fi
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: package-linux-${{ matrix.id }}-${{ hashFiles('cpp/**', 'c_glib/**') }}
+ restore-keys: package-linux-${{ matrix.id }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Prepare apache-arrow-apt-source for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-apt-source/apt
+ for target in *-*; do
+ cp -a ${target} ${target}-arm64
+ done
+ popd
+ - name: Prepare apache-arrow-release for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-release/yum
+ for target in *-*; do
+ cp -a ${target} ${target}-aarch64
+ done
+ popd
+ - name: Update version
+ if: github.ref_type != 'tag'
+ run: |
+ pushd dev/tasks/linux-packages
+ rake version:update
+ popd
+ - name: Login to GitHub Container registry
+ uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 # v3.5.0
+ with:
+ registry: ghcr.io
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+ - name: Wait for creating GitHub Release
+ if: github.ref_type == 'tag'
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ run: |
+ dev/release/utils-watch-gh-workflow.sh \
+ ${GITHUB_REF_NAME} \
+ release_candidate.yml
Review Comment:
https://github.com/apache/arrow/pull/47600/files#diff-96d4591ab159c60b8f65ce7eca019f108423a1e9293123e1336d9251e78c43e6
Here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2362340681
##########
dev/release/04-binary-download.sh:
##########
@@ -39,3 +42,11 @@ crossbow_job_prefix="release-${version_with_rc}"
: ${CROSSBOW_JOB_ID:="${crossbow_job_prefix}-${CROSSBOW_JOB_NUMBER}"}
archery crossbow download-artifacts --no-fetch ${CROSSBOW_JOB_ID} "$@"
+
+# Wait for the GitHub Workflow that creates the Linux packages
+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "linux_packaging.yml"
+
+RUN_ID=$(get_run_id)
+# Download the artifacts created by the linux_packaging.yml workflow
+download_artifacts "${SOURCE_DIR}/../../packages/${CROSSBOW_JOB_ID}"
Review Comment:
The downloaded artifact maintains the repository structure, see:
```
$ tree packages/almalinux-8-amd64/
packages/almalinux-8-amd64/
├── apache-arrow
│ └── yum
│ └── repositories
│ └── almalinux
│ └── 8
│ ├── source
│ │ └── SRPMS
│ │ └── arrow-22.0.0.dev20250919-1.el8.src.rpm
│ └── x86_64
│ └── Packages
│ ├── arrow2200-acero-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-acero-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-compute-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-compute-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-dataset-glib-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-dataset-glib-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-dataset-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-dataset-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-flight-glib-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-flight-glib-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-flight-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-flight-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-flight-sql-glib-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-flight-sql-glib-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-flight-sql-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-flight-sql-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-glib-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-glib-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow2200-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-acero-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-compute-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-dataset-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-dataset-glib-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-dataset-glib-doc-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-debugsource-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-flight-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-flight-glib-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-flight-glib-doc-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-flight-sql-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-flight-sql-glib-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-flight-sql-glib-doc-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-glib-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-glib-doc-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-tools-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── arrow-tools-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── gandiva2200-glib-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── gandiva2200-glib-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── gandiva2200-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── gandiva2200-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── gandiva-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── gandiva-glib-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── gandiva-glib-doc-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── parquet2200-glib-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── parquet2200-glib-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── parquet2200-libs-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── parquet2200-libs-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── parquet-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── parquet-glib-devel-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── parquet-glib-doc-22.0.0.dev20250919-1.el8.x86_64.rpm
│ ├── parquet-tools-22.0.0.dev20250919-1.el8.x86_64.rpm
│ └── parquet-tools-debuginfo-22.0.0.dev20250919-1.el8.x86_64.rpm
└── apache-arrow-release
└── yum
└── repositories
└── almalinux
└── 8
├── source
│ └── SRPMS
│ └── apache-arrow-release-22.0.0.dev20250919-1.el8.src.rpm
└── x86_64
└── Packages
└── apache-arrow-release-22.0.0.dev20250919-1.el8.noarch.rpm
```
but the ones from archery are on a flat folder. I am unsure how the signing of those work, should we flatten them?
```
$ tree packages/nightly-packaging-2025-09-19-0/almalinux-8-amd64/
packages/nightly-packaging-2025-09-19-0/almalinux-8-amd64/
├── apache-arrow-release-22.0.0.dev150-1.el8.noarch.rpm
├── apache-arrow-release-22.0.0.dev150-1.el8.src.rpm
├── arrow2200-acero-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-acero-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-compute-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-compute-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-dataset-glib-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-dataset-glib-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-dataset-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-dataset-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-22.0.0.dev150-1.el8.src.rpm
├── arrow2200-flight-glib-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-flight-glib-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-flight-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-flight-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-flight-sql-glib-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-flight-sql-glib-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-flight-sql-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-flight-sql-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-glib-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-glib-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow2200-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-acero-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-compute-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-dataset-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-dataset-glib-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-dataset-glib-doc-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-debugsource-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-flight-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-flight-glib-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-flight-glib-doc-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-flight-sql-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-flight-sql-glib-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-flight-sql-glib-doc-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-glib-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-glib-doc-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-tools-22.0.0.dev150-1.el8.x86_64.rpm
├── arrow-tools-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── gandiva2200-glib-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── gandiva2200-glib-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── gandiva2200-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── gandiva2200-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── gandiva-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── gandiva-glib-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── gandiva-glib-doc-22.0.0.dev150-1.el8.x86_64.rpm
├── parquet2200-glib-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── parquet2200-glib-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── parquet2200-libs-22.0.0.dev150-1.el8.x86_64.rpm
├── parquet2200-libs-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
├── parquet-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── parquet-glib-devel-22.0.0.dev150-1.el8.x86_64.rpm
├── parquet-glib-doc-22.0.0.dev150-1.el8.x86_64.rpm
├── parquet-tools-22.0.0.dev150-1.el8.x86_64.rpm
└── parquet-tools-debuginfo-22.0.0.dev150-1.el8.x86_64.rpm
1 directory, 57 files
```
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,368 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra: Linux Packaging"
+ parent-workflow: linux_packaging
+
+ package:
+ needs: check-labels
+ name: ${{ matrix.title }}
+ runs-on: ${{ matrix.runs-on }}
+ if: needs.check-labels.outputs.ci-extra == 'true'
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-8
+ title: AlmaLinux 8 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: extra-${{ matrix.target }}-${{ hashFiles('cpp/**') }}
+ restore-keys: extra-${{ matrix.target }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Get Arrow Version
+ id: get-arrow-version
+ run: |
+ # In case of dev (no workflow_dispatch inputs) compute ARROW_VERSION and
+ # use the same version for NO_RC_VERSION.
+ if [ -z "${ARROW_VERSION}" ]; then
+ ARROW_VERSION=$(git describe --tags --abbrev=0 --match "apache-arrow-[0-9]*.*" | sed 's/^apache-arrow-//')
+ ARROW_VERSION="${ARROW_VERSION}$(date +%Y%m%d)"
+ echo "ARROW_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ echo "NO_RC_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ fi
+ - name: Build
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake version:update ARROW_RELEASE_TIME="$(date --iso-8601=seconds)"
+ rake docker:pull || :
+ rake --trace ${{ matrix.task-namespace }}:build BUILD_DIR=build
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ # TODO: Investigate what is this REPO variable and where is coming from
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Login to Dockerhub
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ uses: docker/login-action@v2
+ with:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ - name: Docker Push
+ continue-on-error: true
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ shell: bash
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:push
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Build artifact paths
+ id: artifact-paths
+ shell: bash
+ run: |
+ paths=""
+ for ext in ${{ join(matrix.upload-extensions, ' ') }}; do
+ paths="$paths dev/tasks/linux-packages/*/*/repositories/**/*.${ext}"
+ done
+ echo $paths
+ echo "paths=$paths" >> $GITHUB_OUTPUT
+ - name: Upload the artifacts to the job
+ uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+ with:
+ name: ${{ matrix.target }}-${{ matrix.architecture }}
+ path: ${{ steps.artifact-paths.outputs.paths }}
+ - name: Download Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ path: artifacts-downloaded
+ - name: Expected uploaded yum artifacts
Review Comment:
Should we move this and the next step to individual shell scripts (or python / ruby tests)?
This is only for yum and we will have to do something similar for `.deb` which makes the workflow slightly long to work with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2372325758
##########
dev/release/04-binary-download.sh:
##########
@@ -39,3 +42,22 @@ crossbow_job_prefix="release-${version_with_rc}"
: ${CROSSBOW_JOB_ID:="${crossbow_job_prefix}-${CROSSBOW_JOB_NUMBER}"}
archery crossbow download-artifacts --no-fetch ${CROSSBOW_JOB_ID} "$@"
+
+# Wait for the GitHub Workflow that creates the Linux packages
+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
+
+RUN_ID=$(get_run_id)
+# Download the artifacts created by the package_linux.yml workflow
+download_artifacts "${SOURCE_DIR}/../../packages/${CROSSBOW_JOB_ID}"
Review Comment:
so you mean, for scheduled or PR/push use artifacts but if `workflow_dispatch` was used, which is trigger of RC, upload to release? I don't want to create a release every time the job is executed but that could work. In that case we might use artifacts to test and upload to release if tests are successful and job type was `workflow_dispatch`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2378229933
##########
.github/workflows/package_linux.yml:
##########
@@ -0,0 +1,298 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Package Linux
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ parent-workflow: package_linux
+
+ package:
+ name: ${{ matrix.id }}
+ runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
+ needs: check-labels
+ if: >-
+ needs.check-labels.outputs.force == 'true' ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: Package: Linux')
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ id:
+ - almalinux-8-amd64
+ - almalinux-8-arm64
+ - almalinux-9-amd64
+ - almalinux-9-arm64
+ - almalinux-10-amd64
+ - almalinux-10-arm64
+ - amazon-linux-2023-amd64
+ - amazon-linux-2023-arm64
+ - centos-9-stream-amd64
+ - centos-9-stream-arm64
+ - centos-7-amd64
+ - debian-bookworm-amd64
+ - debian-bookworm-arm64
+ - debian-trixie-amd64
+ - debian-trixie-arm64
+ - debian-forky-amd64
+ - debian-forky-arm64
+ - ubuntu-jammy-amd64
+ - ubuntu-jammy-arm64
+ - ubuntu-noble-amd64
+ - ubuntu-noble-arm64
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Prepare environment variables
+ env:
+ ID: ${{ matrix.id }}
+ run: |
+ set -ex
+ # Example: almalinux-8-amd64 -> amd64
+ architecture="${ID##*-}"
+ echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> almalinux-8
+ target="${ID%-*}"
+ case "${target}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
+ echo "UPLOAD_EXTENSIONS=rpm" >> "${GITHUB_ENV}"
+ if [[ "${architecture}" == "arm64" ]]; then
+ # Example: almalinux-8 -> almalinux-8-aarch64
+ target="${target}-aarch64"
+ fi
+ echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ *)
+ echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
+ upload_extensions=(ddeb deb debian.tar.xz .dsc .orig.tar.gz)
+ echo "UPLOAD_EXTENSIONS=${upload_extensions[*]}" >> "${GITHUB_ENV}"
+ if [[ "${architecture}" == "arm64" ]]; then
+ # Example: ubuntu-noble -> ubuntu-noble-arm64
+ target="${target}-arm64"
+ fi
+ echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ esac
+ echo "TARGET=${target}" >> "${GITHUB_ENV}"
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: package-linux-${{ matrix.id }}-${{ hashFiles('cpp/**') }}
+ restore-keys: package-linux-${{ matrix.id }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Prepare apache-arrow-apt-source for arm64
+ if: ${{ env.ARCHITECTURE == 'arm64' }}
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-apt-source/apt
+ for target in *-*; do
+ cp -a ${target} ${target}-arm64
+ done
+ popd
+ - name: Prepare apache-arrow-release for arm64
+ if: ${{ env.ARCHITECTURE == 'arm64' }}
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-release/yum
+ for target in *-*; do
+ cp -a ${target} ${target}-aarch64
+ done
+ popd
+ - name: Get Arrow Version
+ id: get-arrow-version
+ run: |
+ # In case of dev (no workflow_dispatch inputs) compute ARROW_VERSION and
+ # use the same version for NO_RC_VERSION.
+ if [ -z "${ARROW_VERSION}" ]; then
+ ARROW_VERSION=$(git describe --tags --abbrev=0 --match "apache-arrow-[0-9]*.*" | sed 's/^apache-arrow-//')
+ ARROW_VERSION="${ARROW_VERSION}$(date +%Y%m%d)"
+ echo "ARROW_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ echo "NO_RC_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ fi
+ - name: Build
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake version:update ARROW_RELEASE_TIME="$(date --iso-8601=seconds)"
+ rake docker:pull || :
+ rake --trace ${TASK_NAMESPACE}:build BUILD_DIR=build
+ popd
+ env:
+ REPO: ghcr.io/${{ github.repository }}-package-linux
+ - name: Login to Dockerhub
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 # v3.5.0
+ with:
+ registry: ghcr.io
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN}}
+ - name: Docker Push
+ continue-on-error: true
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ shell: bash
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:push
+ popd
+ env:
+ REPO: ${{ secrets.REPO }}
+ - name: Build artifact tarball
+ shell: bash
+ run: |
+ set -ex
+ pushd dev/tasks/linux-packages
+ tar cvzf ${{ matrix.id }}.tar.gz */${TASK_NAMESPACE}/repositories
Review Comment:
I have tried to simplify the paths slightly more but I am finding quite difficult to understand how all tangles together. How do we generate the repositories from those rpms, how we merge these files with other existing files from the repo, how do we create the testing repositories, etcetera so I am reverting my latest changes. I am happy to leave it like this (current generated structure) and try to simplify on a future PR or if you want to guide me on how to do it or if you want to push to this branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2408582436
##########
.github/workflows/package_linux.yml:
##########
@@ -0,0 +1,313 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Package Linux
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ tags:
+ - "apache-arrow-*-rc*"
+ pull_request:
+ paths:
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ # Upload to GitHub Release
+ contents: write
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ parent-workflow: package_linux
+
+ package:
+ name: ${{ matrix.id }}
+ runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
+ needs: check-labels
+ if: >-
+ needs.check-labels.outputs.force == 'true' ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: Package: Linux')
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ id:
+ - almalinux-8-amd64
+ - almalinux-8-arm64
+ - almalinux-9-amd64
+ - almalinux-9-arm64
+ - almalinux-10-amd64
+ - almalinux-10-arm64
+ - amazon-linux-2023-amd64
+ - amazon-linux-2023-arm64
+ - centos-9-stream-amd64
+ - centos-9-stream-arm64
+ - centos-7-amd64
+ - debian-bookworm-amd64
+ - debian-bookworm-arm64
+ - debian-trixie-amd64
+ - debian-trixie-arm64
+ - debian-forky-amd64
+ - debian-forky-arm64
+ - ubuntu-jammy-amd64
+ - ubuntu-jammy-arm64
+ - ubuntu-noble-amd64
+ - ubuntu-noble-arm64
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Prepare environment variables
+ env:
+ ID: ${{ matrix.id }}
+ run: |
+ set -x
+ case "${ID}" in
+ centos-*)
+ # Example: centos-9-stream-amd64 -> centos
+ distribution="${ID%%-*}"
+ ;;
+ *)
+ # Example: almalinux-8-amd64 -> almalinux
+ # Example: amazon-linux-2023-amd64 -> amazon-linux
+ distribution="${ID%-*-*}"
+ ;;
+ esac
+ echo "DISTRIBUTION=${distribution}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> amd64
+ architecture="${ID##*-}"
+ echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> almalinux-8
+ target="${ID%-*}"
+ case "${target}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: almalinux-8 -> almalinux-8-aarch64
+ target="${target}-aarch64"
+ fi
+ echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ *)
+ echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: ubuntu-noble -> ubuntu-noble-arm64
+ target="${target}-arm64"
+ fi
+ echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ esac
+ echo "TARGET=${target}" >> "${GITHUB_ENV}"
+
+ if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+ # Example: apache-arrow-21.0.0-rc0 -> 21.0.0-rc0
+ version="${GITHUB_REF_NAME#apache-arrow-}"
+ echo "ARROW_VERSION=${version}" >> "${GITHUB_ENV}"
+ fi
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: package-linux-${{ matrix.id }}-${{ hashFiles('cpp/**', 'c_glib/**') }}
+ restore-keys: package-linux-${{ matrix.id }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Prepare apache-arrow-apt-source for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-apt-source/apt
+ for target in *-*; do
+ cp -a ${target} ${target}-arm64
+ done
+ popd
+ - name: Prepare apache-arrow-release for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-release/yum
+ for target in *-*; do
+ cp -a ${target} ${target}-aarch64
+ done
+ popd
+ - name: Update version
+ if: github.ref_type != 'tag'
+ run: |
+ pushd dev/tasks/linux-packages
+ rake version:update
+ popd
+ - name: Login to GitHub Container registry
+ uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 # v3.5.0
+ with:
+ registry: ghcr.io
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+ - name: Wait for creating GitHub Release
+ if: github.ref_type == 'tag'
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ run: |
+ dev/release/utils-watch-gh-workflow.sh \
+ ${GITHUB_REF_NAME} \
+ release_candidate.yml
Review Comment:
Right.
We want to use the source archive uploaded to GitHub release by `release_candidate.yml` when we build Linux packages. So we need to wait for GitHub release here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2397529260
##########
dev/release/binary-task.rb:
##########
@@ -1876,7 +1851,6 @@ def available_yum_targets
["almalinux", "8"],
["amazon-linux", "2023"],
["centos", "9-stream"],
- ["centos", "8-stream"],
Review Comment:
We already dropped support for CentOS Stream 8: https://github.com/apache/arrow/pull/46953
##########
.github/workflows/release_candidate.yml:
##########
@@ -23,9 +23,7 @@ on:
- '**'
- '!dependabot/**'
tags:
- # Trigger workflow when a tag whose name matches the pattern
- # "apache-arrow-{MAJOR}.{MINOR}.{PATCH}-rc{RC_NUM}" is pushed.
- - "apache-arrow-[0-9]+.[0-9]+.[0-9]+-rc[0-9]+"
+ - "apache-arrow-*-rc*"
Review Comment:
Simplify and unify tag patterns.
##########
dev/release/binary-task.rb:
##########
@@ -1649,41 +1649,16 @@ def define_apt_rc_tasks
pool_dir = "#{distribution_dir}/pool/#{code_name}"
rm_rf(pool_dir, verbose: verbose?)
mkdir_p(pool_dir, verbose: verbose?)
+
source_dir_prefix = "#{artifacts_dir}/#{distribution}-#{code_name}"
- # apache/arrow uses debian-bookworm-{amd64,arm64} but
- # apache/arrow-adbc uses debian-bookworm. So the following
+ # apache/arrow uses debian-bookworm-{amd64,arm64}.tar.gz but
+ # apache/arrow-adbc uses debian-bookworm.tar.gz So the following
# glob must much both of them.
- Dir.glob("#{source_dir_prefix}*/*") do |path|
- base_name = File.basename(path)
- package_name = ENV["DEB_PACKAGE_NAME"]
- if package_name.nil? or package_name.empty?
- if base_name.start_with?("apache-arrow-apt-source")
- package_name = "apache-arrow-apt-source"
- else
- package_name = "apache-arrow"
- end
- end
- destination_path = [
- pool_dir,
- component,
- package_name[0],
- package_name,
- base_name,
- ].join("/")
- copy_artifact(path,
- destination_path,
- progress_reporter)
- case base_name
- when /\A[^_]+-apt-source_.*\.deb\z/
- latest_apt_source_package_path = [
- distribution_dir,
- "#{package_name}-latest-#{code_name}.deb"
- ].join("/")
- copy_artifact(path,
- latest_apt_source_package_path,
- progress_reporter)
- end
+ Dir.glob("#{source_dir_prefix}*.tar.gz") do |tar_gz|
+ sh("tar", "xf", tar_gz, "-C", incoming_dir)
+ progress_reporter.advance
end
Review Comment:
We can use this for ADBC too but we need some changes in ADBC: https://github.com/apache/arrow-adbc/issues/3510
##########
dev/tasks/linux-packages/helper.rb:
##########
@@ -72,13 +72,12 @@ def detect_env(name)
raise "Failed to detect #{name} environment variable"
end
- def detect_repo
- detect_env("REPO")
+ def github_repository
+ ENV["GITHUB_REPOSITORY"] || "apache/arrow"
Review Comment:
GitHub Actions provides `GITHUB_REPOSITORY` environment variable by default. So this uses fork's GitHub Container registry on fork automatically.
##########
.github/workflows/package_linux.yml:
##########
@@ -0,0 +1,301 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Package Linux
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - "apache-arrow-*-rc*"
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ # Upload to GitHub Release
+ contents: write
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ parent-workflow: package_linux
+
+ package:
+ name: ${{ matrix.id }}
+ runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
+ needs: check-labels
+ if: >-
+ needs.check-labels.outputs.force == 'true' ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: Package: Linux')
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ id:
+ - almalinux-8-amd64
+ - almalinux-8-arm64
+ - almalinux-9-amd64
+ - almalinux-9-arm64
+ - almalinux-10-amd64
+ - almalinux-10-arm64
+ - amazon-linux-2023-amd64
+ - amazon-linux-2023-arm64
+ - centos-9-stream-amd64
+ - centos-9-stream-arm64
+ - centos-7-amd64
+ - debian-bookworm-amd64
+ - debian-bookworm-arm64
+ - debian-trixie-amd64
+ - debian-trixie-arm64
+ - debian-forky-amd64
+ - debian-forky-arm64
+ - ubuntu-jammy-amd64
+ - ubuntu-jammy-arm64
+ - ubuntu-noble-amd64
+ - ubuntu-noble-arm64
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Prepare environment variables
+ env:
+ ID: ${{ matrix.id }}
+ run: |
+ set -x
+ # Example: almalinux-8-amd64 -> almalinux
+ distribution="${ID%-*-*}"
+ echo "DISTRIBUTION=${distribution}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> amd64
+ architecture="${ID##*-}"
+ echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> almalinux-8
+ target="${ID%-*}"
+ case "${target}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: almalinux-8 -> almalinux-8-aarch64
+ target="${target}-aarch64"
+ fi
+ echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ *)
+ echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: ubuntu-noble -> ubuntu-noble-arm64
+ target="${target}-arm64"
+ fi
+ echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ esac
+ echo "TARGET=${target}" >> "${GITHUB_ENV}"
+
+ if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+ # Example: apache-arrow-21.0.0-rc0 -> 21.0.0-rc0
+ version="${GITHUB_REF_NAME#apache-arrow-}"
+ echo "ARROW_VERSION=${version}" >> "${GITHUB_ENV}"
+ fi
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: package-linux-${{ matrix.id }}-${{ hashFiles('cpp/**', 'c_glib/**') }}
+ restore-keys: package-linux-${{ matrix.id }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Prepare apache-arrow-apt-source for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-apt-source/apt
+ for target in *-*; do
+ cp -a ${target} ${target}-arm64
+ done
+ popd
+ - name: Prepare apache-arrow-release for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-release/yum
+ for target in *-*; do
+ cp -a ${target} ${target}-aarch64
+ done
+ popd
+ - name: Update version
+ if: github.ref_type != 'tag'
+ run: |
+ pushd dev/tasks/linux-packages
+ rake version:update
+ popd
+ - name: Login to GitHub Container registry
+ uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 # v3.5.0
+ with:
+ registry: ghcr.io
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+ - name: Wait for creating GitHub Release
+ if: github.ref_type == 'tag'
+ run: |
+ dev/release/utils-watch-gh-workflow.sh \
+ ${GITHUB_REF_NAME} \
+ release_candidate.yml
+ - name: Build
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:pull || :
+ rake --trace ${TASK_NAMESPACE}:build BUILD_DIR=build
+ popd
+ - name: Docker Push
+ continue-on-error: true
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.ref_name == 'main'
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:push
+ popd
+ - name: Build artifact tarball
+ run: |
+ mkdir -p "${DISTRIBUTION}"
+ cp -a \
+ dev/tasks/linux-packages/*/${TASK_NAMESPACE}/repositories/${DISTRIBUTION}/* \
+ "${DISTRIBUTION}/"
+ # We use latest .deb/.rpm of
+ # apache-arrow-apt-source/apache-arrow-release built for
+ # amd64 because they are architecture independent.
+ if [ "${ARCHITECTURE}" = "amd64" ]; then
+ if [ "${APT_TASK_NAMESPACE}" = "apt" ]; then
+ # Example: debian-bookworm-amd64 -> debian-bookworm
+ code_name="${ID%-*}"
+ # Example: debian-bookworm -> bookworm
+ code_name="${code_name#*-}"
+ # Create
+ # https://packages.apache.org/artifactory/arrow/${DISTRIBUTION}/apache-arrow-apt-source-latest-${code_name}.deb
+ # for easy to install.
+ cp -a \
+ "${DISTRIBUTION}/pool/${code_name}/a/apache-arrow-apt-source/*.deb \
+ "${DISTRIBUTION}/apache-arrow-apt-source-latest-${code_name}.deb"
+ else
+ # Example: amazon-linux-2023-amd64 -> amazon-linux-2023
+ version="${ID%-*}"
+ # Example: amazon-linux-2023 -> 2023
+ version="${version##*-}"
+ # Create
+ # https://packages.apache.org/artifactory/arrow/${DISTRIBUTION}/${version}/apache-arrow-release-latest.rpm
+ # for easy to install.
+ cp -a \
+ "${DISTRIBUTION}/${version}/x86_64/Packages/apache-arrow-release-*.rpm \
+ "${DISTRIBUTION}/${version}/apache-arrow-release-latest.rpm"
+ fi
+ fi
Review Comment:
This is moved from `binary-task.rb`.
We'll create latest `.deb`/`.rpm` here to simplify `binary-task.rb`.
##########
.github/workflows/package_linux.yml:
##########
@@ -0,0 +1,301 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Package Linux
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - "apache-arrow-*-rc*"
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ # Upload to GitHub Release
+ contents: write
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ parent-workflow: package_linux
+
+ package:
+ name: ${{ matrix.id }}
+ runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
+ needs: check-labels
+ if: >-
+ needs.check-labels.outputs.force == 'true' ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: Package: Linux')
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ id:
+ - almalinux-8-amd64
+ - almalinux-8-arm64
+ - almalinux-9-amd64
+ - almalinux-9-arm64
+ - almalinux-10-amd64
+ - almalinux-10-arm64
+ - amazon-linux-2023-amd64
+ - amazon-linux-2023-arm64
+ - centos-9-stream-amd64
+ - centos-9-stream-arm64
+ - centos-7-amd64
+ - debian-bookworm-amd64
+ - debian-bookworm-arm64
+ - debian-trixie-amd64
+ - debian-trixie-arm64
+ - debian-forky-amd64
+ - debian-forky-arm64
+ - ubuntu-jammy-amd64
+ - ubuntu-jammy-arm64
+ - ubuntu-noble-amd64
+ - ubuntu-noble-arm64
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Prepare environment variables
+ env:
+ ID: ${{ matrix.id }}
+ run: |
+ set -x
+ # Example: almalinux-8-amd64 -> almalinux
+ distribution="${ID%-*-*}"
+ echo "DISTRIBUTION=${distribution}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> amd64
+ architecture="${ID##*-}"
+ echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> almalinux-8
+ target="${ID%-*}"
+ case "${target}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: almalinux-8 -> almalinux-8-aarch64
+ target="${target}-aarch64"
+ fi
+ echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ *)
+ echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: ubuntu-noble -> ubuntu-noble-arm64
+ target="${target}-arm64"
+ fi
+ echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ esac
+ echo "TARGET=${target}" >> "${GITHUB_ENV}"
+
+ if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+ # Example: apache-arrow-21.0.0-rc0 -> 21.0.0-rc0
+ version="${GITHUB_REF_NAME#apache-arrow-}"
+ echo "ARROW_VERSION=${version}" >> "${GITHUB_ENV}"
Review Comment:
We can use `21.0.0-rc0` not `21.0.0` here. The `-rc0` part will be removed by https://github.com/apache/arrow/blob/16ceade8f8b6a2720acf028ca3b6352000c60cdb/dev/tasks/linux-packages/package-task.rb#L37-L45 .
##########
dev/tasks/linux-packages/yum/build.sh:
##########
@@ -93,7 +93,7 @@ fi
repositories="/host/repositories"
repository="${repositories}/${distribution}/${distribution_version}"
rpm_dir="${repository}/${architecture}/Packages"
-srpm_dir="${repository}/source/SRPMS"
+srpm_dir="${repository}/Source/Packages"
Review Comment:
AlmaLinux use `Source/Packages`: https://vault.almalinux.org/10.0/BaseOS/Source/Packages/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #47600:
URL: https://github.com/apache/arrow/pull/47600#issuecomment-3374855694
This is ready.
I've fixed problems found by tag test on my fork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2362353973
##########
.github/workflows/cpp_extra.yml:
##########
@@ -75,47 +79,11 @@ permissions:
jobs:
check-labels:
- name: Check labels
- runs-on: ubuntu-latest
- timeout-minutes: 5
- outputs:
- ci-extra: ${{ steps.check.outputs.ci-extra }}
- steps:
- - name: Checkout Arrow
- if: github.event_name == 'pull_request'
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- - name: Check
- id: check
- env:
- GH_TOKEN: ${{ github.token }}
- run: |
- case "${GITHUB_EVENT_NAME}" in
- push|schedule)
- ci_extra=true
- ;;
- pull_request)
- n_ci_extra_labels=$(
- gh pr view ${{ github.event.number }} \
- --jq '.labels[].name | select(. == "CI: Extra")' \
- --json labels \
- --repo ${GITHUB_REPOSITORY} | wc -l)
- if [ "${n_ci_extra_labels}" -eq 1 ]; then
- ci_extra=true
- else
- git fetch origin ${GITHUB_BASE_REF}
- if git diff --stat origin/${GITHUB_BASE_REF}.. | \
- grep \
- --fixed-strings ".github/workflows/cpp_extra.yml" \
- --quiet; then
- ci_extra=true
- else
- ci_extra=false
- fi
- fi
- ;;
- esac
-
- echo "ci-extra=${ci_extra}" >> "${GITHUB_OUTPUT}"
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra"
Review Comment:
I'd like to move these label to either:
- `CI Extra: cpp`
- `CI Extra: CPP`
- `CI: cpp`
- `CI: CPP`
So we can have the others the same like:
- `CI Extra: linux-packaging`
- `CI Extra: Linux Packaging`
- `CI: linux-packaging`
- `CI: Linux Packaging`
What do you think?
##########
.github/workflows/cpp_extra.yml:
##########
@@ -75,47 +79,11 @@ permissions:
jobs:
check-labels:
- name: Check labels
- runs-on: ubuntu-latest
- timeout-minutes: 5
- outputs:
- ci-extra: ${{ steps.check.outputs.ci-extra }}
- steps:
- - name: Checkout Arrow
- if: github.event_name == 'pull_request'
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- - name: Check
- id: check
- env:
- GH_TOKEN: ${{ github.token }}
- run: |
- case "${GITHUB_EVENT_NAME}" in
- push|schedule)
- ci_extra=true
- ;;
- pull_request)
- n_ci_extra_labels=$(
- gh pr view ${{ github.event.number }} \
- --jq '.labels[].name | select(. == "CI: Extra")' \
- --json labels \
- --repo ${GITHUB_REPOSITORY} | wc -l)
- if [ "${n_ci_extra_labels}" -eq 1 ]; then
- ci_extra=true
- else
- git fetch origin ${GITHUB_BASE_REF}
- if git diff --stat origin/${GITHUB_BASE_REF}.. | \
- grep \
- --fixed-strings ".github/workflows/cpp_extra.yml" \
- --quiet; then
- ci_extra=true
- else
- ci_extra=false
- fi
- fi
- ;;
- esac
-
- echo "ci-extra=${ci_extra}" >> "${GITHUB_OUTPUT}"
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra"
Review Comment:
I'd like to move these labels to either:
- `CI Extra: cpp`
- `CI Extra: CPP`
- `CI: cpp`
- `CI: CPP`
So we can have the others the same like:
- `CI Extra: linux-packaging`
- `CI Extra: Linux Packaging`
- `CI: linux-packaging`
- `CI: Linux Packaging`
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2372420386
##########
dev/release/binary-task.rb:
##########
@@ -1645,45 +1645,37 @@ def define_apt_rc_tasks
progress_label = "Copying: #{distribution} #{code_name}"
progress_reporter = ProgressReporter.new(progress_label)
- distribution_dir = "#{incoming_dir}/#{distribution}"
- pool_dir = "#{distribution_dir}/pool/#{code_name}"
- rm_rf(pool_dir, verbose: verbose?)
- mkdir_p(pool_dir, verbose: verbose?)
- source_dir_prefix = "#{artifacts_dir}/#{distribution}-#{code_name}"
- # apache/arrow uses debian-bookworm-{amd64,arm64} but
- # apache/arrow-adbc uses debian-bookworm. So the following
- # glob must much both of them.
- Dir.glob("#{source_dir_prefix}*/*") do |path|
- base_name = File.basename(path)
+ destination_prefix = "#{incoming_dir}/#{distribution}"
Review Comment:
you can push to this branch if needed! that's completely fine with me. I am trying to simplify the folder structure as you suggested so the binary tasks are even simpler. Thanks, it's good that I am learning a little bit more about our linux packages so I can help you maintain those in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2371796442
##########
dev/release/binary-task.rb:
##########
@@ -1645,45 +1645,37 @@ def define_apt_rc_tasks
progress_label = "Copying: #{distribution} #{code_name}"
progress_reporter = ProgressReporter.new(progress_label)
- distribution_dir = "#{incoming_dir}/#{distribution}"
- pool_dir = "#{distribution_dir}/pool/#{code_name}"
- rm_rf(pool_dir, verbose: verbose?)
- mkdir_p(pool_dir, verbose: verbose?)
- source_dir_prefix = "#{artifacts_dir}/#{distribution}-#{code_name}"
- # apache/arrow uses debian-bookworm-{amd64,arm64} but
- # apache/arrow-adbc uses debian-bookworm. So the following
- # glob must much both of them.
Review Comment:
is adbc still using this binary tasks? in that case we probably want to maintain the previous behaviour for the task too?
##########
dev/release/binary-task.rb:
##########
@@ -1645,45 +1645,37 @@ def define_apt_rc_tasks
progress_label = "Copying: #{distribution} #{code_name}"
progress_reporter = ProgressReporter.new(progress_label)
- distribution_dir = "#{incoming_dir}/#{distribution}"
- pool_dir = "#{distribution_dir}/pool/#{code_name}"
- rm_rf(pool_dir, verbose: verbose?)
- mkdir_p(pool_dir, verbose: verbose?)
- source_dir_prefix = "#{artifacts_dir}/#{distribution}-#{code_name}"
- # apache/arrow uses debian-bookworm-{amd64,arm64} but
- # apache/arrow-adbc uses debian-bookworm. So the following
- # glob must much both of them.
- Dir.glob("#{source_dir_prefix}*/*") do |path|
- base_name = File.basename(path)
+ destination_prefix = "#{incoming_dir}/#{distribution}"
Review Comment:
@kou I've never done anything on ruby (I know, I should learn a little :smiley: ). I've tested these tasks locally (`apt:rc:copy` and `yum:rc:copy`) with a small integration test calling rake directly and seems to be working as expected but I would appreciate some help on those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47600:
URL: https://github.com/apache/arrow/pull/47600#issuecomment-3376811986
After merging your PR, Conbench analyzed the 2 benchmarking runs that have been run so far on merge-commit e137a0423fc5ed16cf960e993a4f855946ff8e0c.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/52148058442) has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2362412802
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,368 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra: Linux Packaging"
+ parent-workflow: linux_packaging
+
+ package:
+ needs: check-labels
+ name: ${{ matrix.title }}
+ runs-on: ${{ matrix.runs-on }}
+ if: needs.check-labels.outputs.ci-extra == 'true'
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-8
+ title: AlmaLinux 8 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: extra-${{ matrix.target }}-${{ hashFiles('cpp/**') }}
+ restore-keys: extra-${{ matrix.target }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Get Arrow Version
+ id: get-arrow-version
+ run: |
+ # In case of dev (no workflow_dispatch inputs) compute ARROW_VERSION and
+ # use the same version for NO_RC_VERSION.
+ if [ -z "${ARROW_VERSION}" ]; then
+ ARROW_VERSION=$(git describe --tags --abbrev=0 --match "apache-arrow-[0-9]*.*" | sed 's/^apache-arrow-//')
+ ARROW_VERSION="${ARROW_VERSION}$(date +%Y%m%d)"
+ echo "ARROW_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ echo "NO_RC_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ fi
+ - name: Build
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake version:update ARROW_RELEASE_TIME="$(date --iso-8601=seconds)"
+ rake docker:pull || :
+ rake --trace ${{ matrix.task-namespace }}:build BUILD_DIR=build
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ # TODO: Investigate what is this REPO variable and where is coming from
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Login to Dockerhub
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ uses: docker/login-action@v2
+ with:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ - name: Docker Push
+ continue-on-error: true
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ shell: bash
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:push
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Build artifact paths
+ id: artifact-paths
+ shell: bash
+ run: |
+ paths=""
+ for ext in ${{ join(matrix.upload-extensions, ' ') }}; do
+ paths="$paths dev/tasks/linux-packages/*/*/repositories/**/*.${ext}"
+ done
+ echo $paths
+ echo "paths=$paths" >> $GITHUB_OUTPUT
+ - name: Upload the artifacts to the job
+ uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+ with:
+ name: ${{ matrix.target }}-${{ matrix.architecture }}
+ path: ${{ steps.artifact-paths.outputs.paths }}
+ - name: Download Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ path: artifacts-downloaded
+ - name: Expected uploaded yum artifacts
Review Comment:
we could also split into `linux_packaging_yum.yml` and `linux_packaging_apt.yml` and have two labels:
`CI Extra: yum linux-packaging`, `CI Extra: apt linux-packaging` to reduce the amount of jobs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2362671608
##########
.github/workflows/check_labels.yml:
##########
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+name: Check Labels Reusable
+
+on:
+ workflow_call:
+ inputs:
+ label:
+ description: "Label to check for"
+ required: true
+ type: string
+ parent-workflow:
+ description: "The parent workflow filename (without .yml)"
+ required: true
+ type: string
+ outputs:
+ ci-extra:
+ description: "Whether to run the extra CI"
+ value: ${{ jobs.check-labels.outputs.ci-extra }}
+
+jobs:
+ check-labels:
+ name: Check labels
+ runs-on: ubuntu-latest
+ timeout-minutes: 5
+ outputs:
+ ci-extra: ${{ steps.check.outputs.ci-extra }}
+ steps:
+ - name: Checkout Arrow
+ if: github.event_name == 'pull_request'
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ - name: Check
+ id: check
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: |
+ set -ex
+ case "${GITHUB_EVENT_NAME}" in
+ push|schedule)
+ ci_extra=true
Review Comment:
Separated output may be easier to use:
```suggestion
echo "force=true" >> "${GITHUB_OUTPUT}"
```
##########
.github/workflows/check_labels.yml:
##########
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+name: Check Labels Reusable
+
+on:
+ workflow_call:
+ inputs:
+ label:
+ description: "Label to check for"
+ required: true
+ type: string
+ parent-workflow:
+ description: "The parent workflow filename (without .yml)"
+ required: true
+ type: string
+ outputs:
+ ci-extra:
+ description: "Whether to run the extra CI"
+ value: ${{ jobs.check-labels.outputs.ci-extra }}
+
+jobs:
+ check-labels:
+ name: Check labels
+ runs-on: ubuntu-latest
+ timeout-minutes: 5
+ outputs:
+ ci-extra: ${{ steps.check.outputs.ci-extra }}
+ steps:
+ - name: Checkout Arrow
+ if: github.event_name == 'pull_request'
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ - name: Check
+ id: check
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: |
+ set -ex
+ case "${GITHUB_EVENT_NAME}" in
+ push|schedule)
+ ci_extra=true
+ ;;
+ pull_request)
+ n_ci_extra_labels=$(
+ gh pr view ${{ github.event.number }} \
+ --jq '.labels[].name | select(. == "${{ inputs.label }}")' \
+ --json labels \
+ --repo ${GITHUB_REPOSITORY} | wc -l)
+ if [ "${n_ci_extra_labels}" -eq 1 ]; then
+ ci_extra=true
+ else
+ git fetch origin ${GITHUB_BASE_REF}
+ git diff --stat origin/${GITHUB_BASE_REF}..
+ if git diff --stat origin/${GITHUB_BASE_REF}.. | \
+ grep \
+ --fixed-strings ".github/workflows/${{ inputs.parent-workflow }}.yml" \
+ --quiet; then
+ ci_extra=true
Review Comment:
```suggestion
echo "force=true" >> "${GITHUB_OUTPUT}"
```
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,435 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra: Linux Packaging"
+ parent-workflow: linux_packaging
+
+ package:
+ needs: check-labels
+ name: ${{ matrix.title }}
+ runs-on: ${{ matrix.runs-on }}
+ if: needs.check-labels.outputs.ci-extra == 'true'
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-8
+ title: AlmaLinux 8 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-8
+ title: AlmaLinux 8 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-9
+ title: AlmaLinux 9 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-9
+ title: AlmaLinux 9 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-10
+ title: AlmaLinux 10 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-10
+ title: AlmaLinux 10 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-7
+ title: CentOS 7 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-9-stream
+ title: CentOS 9 Stream AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: centos-9-stream
+ title: CentOS 9 Stream ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
Review Comment:
We can simplify this:
```suggestion
id:
- almalinux-8-amd64
- almalinux-8-arm64
- almalinux-9-amd64
- almalinux-9-arm64
- almalinux-10-amd64
- almalinux-10-arm64
...
```
```yaml
title: ${{ matrix.id }}
runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
```
```yaml
name: Prepare environment variables
env:
ID: ${{ matrix.id }}
run: |
# almalinux-8-amd64 -> amd64
architecture="${ID##*-}"
echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
# almalinux-8-amd64 -> almalinux-8
target="${ID%-*}"
case "${target}" in
almalinux-*|amazon-linux-*|centos-*)
echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
;;
*)
echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
;;
esac
```
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,435 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
Review Comment:
We don't need "Extra" here because there is no "Extra" version.
##########
.github/workflows/cpp_extra.yml:
##########
@@ -75,47 +79,11 @@ permissions:
jobs:
check-labels:
- name: Check labels
- runs-on: ubuntu-latest
- timeout-minutes: 5
- outputs:
- ci-extra: ${{ steps.check.outputs.ci-extra }}
- steps:
- - name: Checkout Arrow
- if: github.event_name == 'pull_request'
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- - name: Check
- id: check
- env:
- GH_TOKEN: ${{ github.token }}
- run: |
- case "${GITHUB_EVENT_NAME}" in
- push|schedule)
- ci_extra=true
- ;;
- pull_request)
- n_ci_extra_labels=$(
- gh pr view ${{ github.event.number }} \
- --jq '.labels[].name | select(. == "CI: Extra")' \
- --json labels \
- --repo ${GITHUB_REPOSITORY} | wc -l)
- if [ "${n_ci_extra_labels}" -eq 1 ]; then
- ci_extra=true
- else
- git fetch origin ${GITHUB_BASE_REF}
- if git diff --stat origin/${GITHUB_BASE_REF}.. | \
- grep \
- --fixed-strings ".github/workflows/cpp_extra.yml" \
- --quiet; then
- ci_extra=true
- else
- ci_extra=false
- fi
- fi
- ;;
- esac
-
- echo "ci-extra=${ci_extra}" >> "${GITHUB_OUTPUT}"
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra"
Review Comment:
I like `CI: Extra: C++`/`CI: Extra: Package: Linux` or `CI: cpp`/`CI: package: linux`/`CI: package: linux: apt`.
##########
.github/workflows/check_labels.yml:
##########
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+name: Check Labels Reusable
Review Comment:
```suggestion
name: Check Labels Reusable
```
##########
.github/workflows/check_labels.yml:
##########
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+name: Check Labels Reusable
+
+on:
+ workflow_call:
+ inputs:
+ label:
+ description: "Label to check for"
+ required: true
+ type: string
+ parent-workflow:
+ description: "The parent workflow filename (without .yml)"
+ required: true
+ type: string
+ outputs:
+ ci-extra:
+ description: "Whether to run the extra CI"
+ value: ${{ jobs.check-labels.outputs.ci-extra }}
+
+jobs:
+ check-labels:
+ name: Check labels
+ runs-on: ubuntu-latest
+ timeout-minutes: 5
+ outputs:
+ ci-extra: ${{ steps.check.outputs.ci-extra }}
+ steps:
+ - name: Checkout Arrow
+ if: github.event_name == 'pull_request'
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ - name: Check
+ id: check
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: |
+ set -ex
+ case "${GITHUB_EVENT_NAME}" in
+ push|schedule)
+ ci_extra=true
+ ;;
+ pull_request)
+ n_ci_extra_labels=$(
+ gh pr view ${{ github.event.number }} \
+ --jq '.labels[].name | select(. == "${{ inputs.label }}")' \
+ --json labels \
+ --repo ${GITHUB_REPOSITORY} | wc -l)
Review Comment:
How about detecting all `CI: Extra` labels?
```suggestion
{
echo "ci-extra-labels<<LABELS"
gh pr view ${{ github.event.number }} \
--jq '[.labels[].name | select(startswith("CI: Extra"))]' \
--json labels \
--repo ${GITHUB_REPOSITORY}
echo "LABELS"
} >> "${GITHUB_OUTPUT}"
```
We can use it in parent workflow by:
```yaml
if: >-
needs.check-labels.outputs.force == 'true' ||
fromJSON(needs.check-labels.outputs.ci-extra-labels, 'CI: Extra') ||
fromJSON(needs.check-labels.outputs.ci-extra-labels, 'CI: Extra: C++')
```
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,435 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra: Linux Packaging"
+ parent-workflow: linux_packaging
+
+ package:
+ needs: check-labels
+ name: ${{ matrix.title }}
+ runs-on: ${{ matrix.runs-on }}
+ if: needs.check-labels.outputs.ci-extra == 'true'
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-8
+ title: AlmaLinux 8 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-8
+ title: AlmaLinux 8 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-9
+ title: AlmaLinux 9 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-9
+ title: AlmaLinux 9 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-10
+ title: AlmaLinux 10 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-10
+ title: AlmaLinux 10 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-7
+ title: CentOS 7 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-9-stream
+ title: CentOS 9 Stream AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: centos-9-stream
+ title: CentOS 9 Stream ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: extra-${{ matrix.target }}-${{ hashFiles('cpp/**') }}
+ restore-keys: extra-${{ matrix.target }}-
Review Comment:
```suggestion
key: package-linux-${{ matrix.target }}-${{ hashFiles('cpp/**') }}
restore-keys: package-linux-${{ matrix.target }}-
```
##########
.github/workflows/linux_packaging.yml:
##########
Review Comment:
How about `package_linux.yml`? We may add `package_conan.yml`, `package_homeber.yml` and so on later.
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,435 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra: Linux Packaging"
+ parent-workflow: linux_packaging
+
+ package:
+ needs: check-labels
+ name: ${{ matrix.title }}
+ runs-on: ${{ matrix.runs-on }}
+ if: needs.check-labels.outputs.ci-extra == 'true'
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-8
+ title: AlmaLinux 8 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-8
+ title: AlmaLinux 8 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-9
+ title: AlmaLinux 9 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-9
+ title: AlmaLinux 9 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-10
+ title: AlmaLinux 10 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-10
+ title: AlmaLinux 10 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-7
+ title: CentOS 7 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-9-stream
+ title: CentOS 9 Stream AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: centos-9-stream
+ title: CentOS 9 Stream ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: extra-${{ matrix.target }}-${{ hashFiles('cpp/**') }}
+ restore-keys: extra-${{ matrix.target }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Get Arrow Version
+ id: get-arrow-version
+ run: |
+ # In case of dev (no workflow_dispatch inputs) compute ARROW_VERSION and
+ # use the same version for NO_RC_VERSION.
+ if [ -z "${ARROW_VERSION}" ]; then
+ ARROW_VERSION=$(git describe --tags --abbrev=0 --match "apache-arrow-[0-9]*.*" | sed 's/^apache-arrow-//')
+ ARROW_VERSION="${ARROW_VERSION}$(date +%Y%m%d)"
+ echo "ARROW_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ echo "NO_RC_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ fi
+ - name: Build
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake version:update ARROW_RELEASE_TIME="$(date --iso-8601=seconds)"
+ rake docker:pull || :
+ rake --trace ${{ matrix.task-namespace }}:build BUILD_DIR=build
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ # TODO: Investigate what is this REPO variable and where is coming from
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Login to Dockerhub
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ uses: docker/login-action@v2
+ with:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
Review Comment:
If we use `ghcr.io`, we can use `${{ github.actor }}` and `${{ secrets.GITHUB_TOKEN}}` for `username` and `password`.
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,435 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra: Linux Packaging"
+ parent-workflow: linux_packaging
+
+ package:
+ needs: check-labels
+ name: ${{ matrix.title }}
+ runs-on: ${{ matrix.runs-on }}
+ if: needs.check-labels.outputs.ci-extra == 'true'
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-8
+ title: AlmaLinux 8 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-8
+ title: AlmaLinux 8 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-9
+ title: AlmaLinux 9 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-9
+ title: AlmaLinux 9 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-10
+ title: AlmaLinux 10 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-10
+ title: AlmaLinux 10 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-7
+ title: CentOS 7 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-9-stream
+ title: CentOS 9 Stream AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: centos-9-stream
+ title: CentOS 9 Stream ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: extra-${{ matrix.target }}-${{ hashFiles('cpp/**') }}
+ restore-keys: extra-${{ matrix.target }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Get Arrow Version
+ id: get-arrow-version
+ run: |
+ # In case of dev (no workflow_dispatch inputs) compute ARROW_VERSION and
+ # use the same version for NO_RC_VERSION.
+ if [ -z "${ARROW_VERSION}" ]; then
+ ARROW_VERSION=$(git describe --tags --abbrev=0 --match "apache-arrow-[0-9]*.*" | sed 's/^apache-arrow-//')
+ ARROW_VERSION="${ARROW_VERSION}$(date +%Y%m%d)"
+ echo "ARROW_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ echo "NO_RC_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ fi
+ - name: Build
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake version:update ARROW_RELEASE_TIME="$(date --iso-8601=seconds)"
+ rake docker:pull || :
+ rake --trace ${{ matrix.task-namespace }}:build BUILD_DIR=build
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ # TODO: Investigate what is this REPO variable and where is coming from
+ REPO: ${{ secrets.REPO }}
Review Comment:
This is for the Docker image name part. We can use `ghcr.io/${{ github.repository }}-package-linux` here:
```suggestion
REPO: ghcr.io/${{ github.repository }}-package-linux
```
BTW, could you improve method name?
```diff
diff --git a/dev/tasks/linux-packages/helper.rb b/dev/tasks/linux-packages/helper.rb
index 0354188046..ee132f12e9 100644
--- a/dev/tasks/linux-packages/helper.rb
+++ b/dev/tasks/linux-packages/helper.rb
@@ -72,13 +72,13 @@ module Helper
raise "Failed to detect #{name} environment variable"
end
- def detect_repo
+ def docker_image_name
detect_env("REPO")
end
def docker_image(os, architecture)
architecture ||= "amd64"
- "#{detect_repo}:#{architecture}-#{os}-package-#{@package}"
+ "#{docker_image_name}:#{architecture}-#{os}-package-#{@package}"
end
end
end
```
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,435 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra: Linux Packaging"
+ parent-workflow: linux_packaging
+
+ package:
+ needs: check-labels
+ name: ${{ matrix.title }}
+ runs-on: ${{ matrix.runs-on }}
+ if: needs.check-labels.outputs.ci-extra == 'true'
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-8
+ title: AlmaLinux 8 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-8
+ title: AlmaLinux 8 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-9
+ title: AlmaLinux 9 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-9
+ title: AlmaLinux 9 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-10
+ title: AlmaLinux 10 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-10
+ title: AlmaLinux 10 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-7
+ title: CentOS 7 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-9-stream
+ title: CentOS 9 Stream AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: centos-9-stream
+ title: CentOS 9 Stream ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: extra-${{ matrix.target }}-${{ hashFiles('cpp/**') }}
+ restore-keys: extra-${{ matrix.target }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Get Arrow Version
+ id: get-arrow-version
+ run: |
+ # In case of dev (no workflow_dispatch inputs) compute ARROW_VERSION and
+ # use the same version for NO_RC_VERSION.
+ if [ -z "${ARROW_VERSION}" ]; then
+ ARROW_VERSION=$(git describe --tags --abbrev=0 --match "apache-arrow-[0-9]*.*" | sed 's/^apache-arrow-//')
+ ARROW_VERSION="${ARROW_VERSION}$(date +%Y%m%d)"
+ echo "ARROW_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ echo "NO_RC_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ fi
+ - name: Build
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake version:update ARROW_RELEASE_TIME="$(date --iso-8601=seconds)"
+ rake docker:pull || :
+ rake --trace ${{ matrix.task-namespace }}:build BUILD_DIR=build
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ # TODO: Investigate what is this REPO variable and where is coming from
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Login to Dockerhub
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ uses: docker/login-action@v2
+ with:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ - name: Docker Push
+ continue-on-error: true
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ shell: bash
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:push
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Build artifact paths
+ id: artifact-paths
+ shell: bash
+ run: |
+ paths=""
+ for ext in ${{ join(matrix.upload-extensions, ' ') }}; do
+ paths="$paths dev/tasks/linux-packages/*/*/repositories/**/*.${ext}"
+ done
+ echo $paths
+ echo "paths=$paths" >> $GITHUB_OUTPUT
Review Comment:
We can upload all files under `dev/tasks/linux-packages/*/*/repositories`:
```suggestion
pushd dev/tasks/linux-packages
tar cvzf ${{ matrix.id }}.tar.gz */${{ matrix.task_namespace }}/repositories
mv ${{ matrix.id }}.tar.gz # We can upload only this tar.gz
popd
```
##########
.github/workflows/report_ci.yml:
##########
@@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+name: Report CI results
Review Comment:
```suggestion
name: Report CI results
```
##########
dev/release/04-binary-download.sh:
##########
@@ -39,3 +42,11 @@ crossbow_job_prefix="release-${version_with_rc}"
: ${CROSSBOW_JOB_ID:="${crossbow_job_prefix}-${CROSSBOW_JOB_NUMBER}"}
archery crossbow download-artifacts --no-fetch ${CROSSBOW_JOB_ID} "$@"
+
+# Wait for the GitHub Workflow that creates the Linux packages
+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "linux_packaging.yml"
+
+RUN_ID=$(get_run_id)
+# Download the artifacts created by the linux_packaging.yml workflow
+download_artifacts "${SOURCE_DIR}/../../packages/${CROSSBOW_JOB_ID}"
Review Comment:
Not flat folder is better. The current flat folder is a limitation of Crossbow.
Our binary upload script reconstruct the original repository structure from a flat folder: https://github.com/apache/arrow/blob/7ef564884547e9de6d34ccd25d3bc6992111b92c/dev/release/binary-task.rb#L1666-L1675
If we can keep the repository structure, we can simplify the binary upload script.
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,435 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra: Linux Packaging"
+ parent-workflow: linux_packaging
+
+ package:
+ needs: check-labels
+ name: ${{ matrix.title }}
+ runs-on: ${{ matrix.runs-on }}
+ if: needs.check-labels.outputs.ci-extra == 'true'
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-8
+ title: AlmaLinux 8 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-8
+ title: AlmaLinux 8 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-9
+ title: AlmaLinux 9 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-9
+ title: AlmaLinux 9 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-10
+ title: AlmaLinux 10 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-10
+ title: AlmaLinux 10 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-7
+ title: CentOS 7 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-9-stream
+ title: CentOS 9 Stream AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: centos-9-stream
+ title: CentOS 9 Stream ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: extra-${{ matrix.target }}-${{ hashFiles('cpp/**') }}
+ restore-keys: extra-${{ matrix.target }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Get Arrow Version
+ id: get-arrow-version
+ run: |
+ # In case of dev (no workflow_dispatch inputs) compute ARROW_VERSION and
+ # use the same version for NO_RC_VERSION.
+ if [ -z "${ARROW_VERSION}" ]; then
+ ARROW_VERSION=$(git describe --tags --abbrev=0 --match "apache-arrow-[0-9]*.*" | sed 's/^apache-arrow-//')
+ ARROW_VERSION="${ARROW_VERSION}$(date +%Y%m%d)"
+ echo "ARROW_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ echo "NO_RC_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ fi
+ - name: Build
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake version:update ARROW_RELEASE_TIME="$(date --iso-8601=seconds)"
+ rake docker:pull || :
+ rake --trace ${{ matrix.task-namespace }}:build BUILD_DIR=build
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ # TODO: Investigate what is this REPO variable and where is coming from
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Login to Dockerhub
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ uses: docker/login-action@v2
+ with:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ - name: Docker Push
+ continue-on-error: true
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ shell: bash
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:push
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Build artifact paths
+ id: artifact-paths
+ shell: bash
+ run: |
+ paths=""
+ for ext in ${{ join(matrix.upload-extensions, ' ') }}; do
+ paths="$paths dev/tasks/linux-packages/*/*/repositories/**/*.${ext}"
+ done
+ echo $paths
+ echo "paths=$paths" >> $GITHUB_OUTPUT
+ - name: Upload the artifacts to the job
+ uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+ with:
+ name: ${{ matrix.target }}-${{ matrix.architecture }}
+ path: ${{ steps.artifact-paths.outputs.paths }}
+ - name: Download Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ path: artifacts-downloaded
+ - name: Expected uploaded yum artifacts
+ if: matrix.task-namespace == 'yum'
+ id: expected-yum-artifacts
+ run: |
+ set -ex
+ artifacts="
+ apache-arrow-release-${NO_RC_VERSION}-1.[a-z0-9]+.noarch.rpm
+ apache-arrow-release-${NO_RC_VERSION}-1.[a-z0-9]+.src.rpm
+ arrow-${NO_RC_VERSION}-1.[a-z0-9]+.src.rpm
+ arrow[0-9]+-acero-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-acero-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-compute-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-compute-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-dataset-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-dataset-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-dataset-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-dataset-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-dataset-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-tools-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet[0-9]+-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet[0-9]+-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet-tools-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ "
+ if [ "${{ matrix.target }}" != "centos-7" ]; then
+ artifacts="$artifacts
+ arrow[0-9]+-acero-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-compute-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-dataset-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-dataset-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-debugsource-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-sql-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-sql-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-sql-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-sql-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-sql-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-sql-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-sql-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-tools-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva[0-9]+-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva[0-9]+-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva[0-9]+-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva[0-9]+-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet[0-9]+-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet[0-9]+-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet-tools-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm"
+ fi
+ echo $artifacts
+ {
+ echo 'artifacts<<EOF'
+ echo "$artifacts"
+ echo 'EOF'
+ } >> $GITHUB_OUTPUT
+ - name: Validate uploaded yum artifacts
+ if: matrix.task-namespace == 'yum'
+ id: validate-artifacts
+ shell: bash
+ run: |
+ expected_patterns="${{ steps.expected-yum-artifacts.outputs.artifacts }}"
+
+ # Find all RPM files in the extracted directory
+ found_files=$(find artifacts-downloaded/${{ matrix.target }}-${{ matrix.architecture }} -type f -name '*.rpm' | sort)
+
+ # Prepare expected patterns for matching
+ echo "$expected_patterns" | grep -v '^\s*$' > expected_patterns.txt
+
+ # Check for missing artifacts
+ missing=""
+ for pattern in $(cat expected_patterns.txt); do
+ # Escape dots for grep
+ grep_pattern=$(echo "$pattern" | sed 's/\./\\./g')
+ match=$(echo "$found_files" | grep -E "$grep_pattern")
+ if [ -z "$match" ]; then
+ missing="$missing\n$pattern"
+ fi
+ done
+
+ # Check for unexpected artifacts
+ unexpected=""
+ for file in $found_files; do
+ matched=false
+ for pattern in $(cat expected_patterns.txt); do
+ grep_pattern=$(echo "$pattern" | sed 's/\./\\./g')
+ if echo "$file" | grep -qE "$grep_pattern"; then
+ matched=true
+ break
+ fi
+ done
+ if [ "$matched" = false ]; then
+ unexpected="$unexpected\n$file"
+ fi
+ done
+
+ if [ -n "$missing" ]; then
+ echo "Missing expected artifacts:"
+ echo -e "$missing"
+ exit 1
+ fi
+
+ if [ -n "$unexpected" ]; then
+ echo "Unexpected artifacts found:"
+ echo -e "$unexpected"
+ exit 1
+ fi
+
+ echo "All expected artifacts are present, and no unexpected artifacts found."
+ - name: Set up test
+ run: |
+ sudo apt install -y \
+ apt-utils \
+ cpio \
+ createrepo-c \
+ devscripts \
+ gpg \
+ rpm \
+ rsync
+ gem install --user-install apt-dists-merge
+ (echo "Key-Type: RSA"; \
+ echo "Key-Length: 4096"; \
+ echo "Name-Real: Test"; \
+ echo "Name-Email: test@example.com"; \
+ echo "%no-protection") | \
+ gpg --full-generate-key --batch
+ GPG_KEY_ID=$(gpg --list-keys --with-colon test@example.com | grep fpr | cut -d: -f10)
+ echo "GPG_KEY_ID=${GPG_KEY_ID}" >> ${GITHUB_ENV}
+ case "${{ matrix.target }}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ repositories_dir=dev/tasks/linux-packages/apache-arrow-release/yum/repositories
+ rpm2cpio ${repositories_dir}/*/*/*/Packages/apache-arrow-release-*.rpm | \
+ cpio -id
+ mv etc/pki/rpm-gpg/RPM-GPG-KEY-Apache-Arrow \
+ dev/tasks/linux-packages/KEYS
+ ;;
+ esac
+ gpg --export --armor test@example.com >> dev/tasks/linux-packages/KEYS
+ - name: Test
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake --trace ${{ matrix.task-namespace }}:test
+ rm -rf ${{ matrix.task-namespace }}/repositories
Review Comment:
This is needless.
```suggestion
```
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,368 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra: Linux Packaging"
+ parent-workflow: linux_packaging
+
+ package:
+ needs: check-labels
+ name: ${{ matrix.title }}
+ runs-on: ${{ matrix.runs-on }}
+ if: needs.check-labels.outputs.ci-extra == 'true'
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-8
+ title: AlmaLinux 8 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: extra-${{ matrix.target }}-${{ hashFiles('cpp/**') }}
+ restore-keys: extra-${{ matrix.target }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Get Arrow Version
+ id: get-arrow-version
+ run: |
+ # In case of dev (no workflow_dispatch inputs) compute ARROW_VERSION and
+ # use the same version for NO_RC_VERSION.
+ if [ -z "${ARROW_VERSION}" ]; then
+ ARROW_VERSION=$(git describe --tags --abbrev=0 --match "apache-arrow-[0-9]*.*" | sed 's/^apache-arrow-//')
+ ARROW_VERSION="${ARROW_VERSION}$(date +%Y%m%d)"
+ echo "ARROW_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ echo "NO_RC_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ fi
+ - name: Build
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake version:update ARROW_RELEASE_TIME="$(date --iso-8601=seconds)"
+ rake docker:pull || :
+ rake --trace ${{ matrix.task-namespace }}:build BUILD_DIR=build
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ # TODO: Investigate what is this REPO variable and where is coming from
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Login to Dockerhub
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ uses: docker/login-action@v2
+ with:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ - name: Docker Push
+ continue-on-error: true
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ shell: bash
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:push
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Build artifact paths
+ id: artifact-paths
+ shell: bash
+ run: |
+ paths=""
+ for ext in ${{ join(matrix.upload-extensions, ' ') }}; do
+ paths="$paths dev/tasks/linux-packages/*/*/repositories/**/*.${ext}"
+ done
+ echo $paths
+ echo "paths=$paths" >> $GITHUB_OUTPUT
+ - name: Upload the artifacts to the job
+ uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+ with:
+ name: ${{ matrix.target }}-${{ matrix.architecture }}
+ path: ${{ steps.artifact-paths.outputs.paths }}
+ - name: Download Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ path: artifacts-downloaded
+ - name: Expected uploaded yum artifacts
Review Comment:
Ah, we don't need to validate target files. Our tests will detect missing files.
##########
.github/workflows/linux_packaging.yml:
##########
@@ -0,0 +1,435 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Linux Packaging Extra
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/linux_packaging.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra: Linux Packaging"
+ parent-workflow: linux_packaging
+
+ package:
+ needs: check-labels
+ name: ${{ matrix.title }}
+ runs-on: ${{ matrix.runs-on }}
+ if: needs.check-labels.outputs.ci-extra == 'true'
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ include:
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-8
+ title: AlmaLinux 8 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-8
+ title: AlmaLinux 8 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-9
+ title: AlmaLinux 9 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-9
+ title: AlmaLinux 9 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: almalinux-10
+ title: AlmaLinux 10 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: almalinux-10
+ title: AlmaLinux 10 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: amazon-linux-2023
+ title: Amazon Linux 2023 ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-7
+ title: CentOS 7 AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: amd64
+ runs-on: ubuntu-latest
+ target: centos-9-stream
+ title: CentOS 9 Stream AMD64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ - architecture: arm64
+ runs-on: ubuntu-24.04-arm
+ target: centos-9-stream
+ title: CentOS 9 Stream ARM64
+ task-namespace: yum
+ upload-extensions:
+ - rpm
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: extra-${{ matrix.target }}-${{ hashFiles('cpp/**') }}
+ restore-keys: extra-${{ matrix.target }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Get Arrow Version
+ id: get-arrow-version
+ run: |
+ # In case of dev (no workflow_dispatch inputs) compute ARROW_VERSION and
+ # use the same version for NO_RC_VERSION.
+ if [ -z "${ARROW_VERSION}" ]; then
+ ARROW_VERSION=$(git describe --tags --abbrev=0 --match "apache-arrow-[0-9]*.*" | sed 's/^apache-arrow-//')
+ ARROW_VERSION="${ARROW_VERSION}$(date +%Y%m%d)"
+ echo "ARROW_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ echo "NO_RC_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ fi
+ - name: Build
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake version:update ARROW_RELEASE_TIME="$(date --iso-8601=seconds)"
+ rake docker:pull || :
+ rake --trace ${{ matrix.task-namespace }}:build BUILD_DIR=build
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ # TODO: Investigate what is this REPO variable and where is coming from
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Login to Dockerhub
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ uses: docker/login-action@v2
+ with:
+ username: ${{ secrets.DOCKERHUB_USER }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ - name: Docker Push
+ continue-on-error: true
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ shell: bash
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:push
+ popd
+ env:
+ APT_TARGETS: ${{ matrix.target }}
+ REPO: ${{ secrets.REPO }}
+ YUM_TARGETS: ${{ matrix.target }}
+ - name: Build artifact paths
+ id: artifact-paths
+ shell: bash
+ run: |
+ paths=""
+ for ext in ${{ join(matrix.upload-extensions, ' ') }}; do
+ paths="$paths dev/tasks/linux-packages/*/*/repositories/**/*.${ext}"
+ done
+ echo $paths
+ echo "paths=$paths" >> $GITHUB_OUTPUT
+ - name: Upload the artifacts to the job
+ uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+ with:
+ name: ${{ matrix.target }}-${{ matrix.architecture }}
+ path: ${{ steps.artifact-paths.outputs.paths }}
+ - name: Download Artifacts
+ uses: actions/download-artifact@v4
+ with:
+ path: artifacts-downloaded
+ - name: Expected uploaded yum artifacts
+ if: matrix.task-namespace == 'yum'
+ id: expected-yum-artifacts
+ run: |
+ set -ex
+ artifacts="
+ apache-arrow-release-${NO_RC_VERSION}-1.[a-z0-9]+.noarch.rpm
+ apache-arrow-release-${NO_RC_VERSION}-1.[a-z0-9]+.src.rpm
+ arrow-${NO_RC_VERSION}-1.[a-z0-9]+.src.rpm
+ arrow[0-9]+-acero-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-acero-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-compute-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-compute-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-dataset-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-dataset-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-dataset-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-dataset-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-dataset-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-tools-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet[0-9]+-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet[0-9]+-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet-tools-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ "
+ if [ "${{ matrix.target }}" != "centos-7" ]; then
+ artifacts="$artifacts
+ arrow[0-9]+-acero-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-compute-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-dataset-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-dataset-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-debugsource-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-sql-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-sql-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-flight-sql-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-sql-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-sql-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-sql-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-flight-sql-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow[0-9]+-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ arrow-tools-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva-glib-devel-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva-glib-doc-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva[0-9]+-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva[0-9]+-glib-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva[0-9]+-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ gandiva[0-9]+-libs-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet[0-9]+-glib-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet[0-9]+-libs-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm
+ parquet-tools-debuginfo-${NO_RC_VERSION}-1.[a-z0-9]+.[a-z0-9_]+.rpm"
+ fi
+ echo $artifacts
+ {
+ echo 'artifacts<<EOF'
+ echo "$artifacts"
+ echo 'EOF'
+ } >> $GITHUB_OUTPUT
+ - name: Validate uploaded yum artifacts
+ if: matrix.task-namespace == 'yum'
+ id: validate-artifacts
+ shell: bash
+ run: |
+ expected_patterns="${{ steps.expected-yum-artifacts.outputs.artifacts }}"
+
+ # Find all RPM files in the extracted directory
+ found_files=$(find artifacts-downloaded/${{ matrix.target }}-${{ matrix.architecture }} -type f -name '*.rpm' | sort)
+
+ # Prepare expected patterns for matching
+ echo "$expected_patterns" | grep -v '^\s*$' > expected_patterns.txt
+
+ # Check for missing artifacts
+ missing=""
+ for pattern in $(cat expected_patterns.txt); do
+ # Escape dots for grep
+ grep_pattern=$(echo "$pattern" | sed 's/\./\\./g')
+ match=$(echo "$found_files" | grep -E "$grep_pattern")
+ if [ -z "$match" ]; then
+ missing="$missing\n$pattern"
+ fi
+ done
+
+ # Check for unexpected artifacts
+ unexpected=""
+ for file in $found_files; do
+ matched=false
+ for pattern in $(cat expected_patterns.txt); do
+ grep_pattern=$(echo "$pattern" | sed 's/\./\\./g')
+ if echo "$file" | grep -qE "$grep_pattern"; then
+ matched=true
+ break
+ fi
+ done
+ if [ "$matched" = false ]; then
+ unexpected="$unexpected\n$file"
+ fi
+ done
+
+ if [ -n "$missing" ]; then
+ echo "Missing expected artifacts:"
+ echo -e "$missing"
+ exit 1
+ fi
+
+ if [ -n "$unexpected" ]; then
+ echo "Unexpected artifacts found:"
+ echo -e "$unexpected"
+ exit 1
+ fi
+
+ echo "All expected artifacts are present, and no unexpected artifacts found."
+ - name: Set up test
+ run: |
+ sudo apt install -y \
Review Comment:
```suggestion
pushd dev/tasks/linux-packages
rm -rf */${{ matrix.task-namespace }}/repositories # Remove artifacts
tar xf ${{ matrix.id }}.tar.gz # Use uploaded artifacts
popd
sudo apt install -y \
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou merged PR #47600:
URL: https://github.com/apache/arrow/pull/47600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2409693484
##########
.github/workflows/package_linux.yml:
##########
@@ -0,0 +1,313 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Package Linux
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ tags:
+ - "apache-arrow-*-rc*"
+ pull_request:
+ paths:
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ # Upload to GitHub Release
+ contents: write
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ parent-workflow: package_linux
+
+ package:
+ name: ${{ matrix.id }}
+ runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
+ needs: check-labels
+ if: >-
+ needs.check-labels.outputs.force == 'true' ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: Package: Linux')
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ id:
+ - almalinux-8-amd64
+ - almalinux-8-arm64
+ - almalinux-9-amd64
+ - almalinux-9-arm64
+ - almalinux-10-amd64
+ - almalinux-10-arm64
+ - amazon-linux-2023-amd64
+ - amazon-linux-2023-arm64
+ - centos-9-stream-amd64
+ - centos-9-stream-arm64
+ - centos-7-amd64
+ - debian-bookworm-amd64
+ - debian-bookworm-arm64
+ - debian-trixie-amd64
+ - debian-trixie-arm64
+ - debian-forky-amd64
+ - debian-forky-arm64
+ - ubuntu-jammy-amd64
+ - ubuntu-jammy-arm64
+ - ubuntu-noble-amd64
+ - ubuntu-noble-arm64
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Prepare environment variables
+ env:
+ ID: ${{ matrix.id }}
+ run: |
+ set -x
+ case "${ID}" in
+ centos-*)
+ # Example: centos-9-stream-amd64 -> centos
+ distribution="${ID%%-*}"
+ ;;
+ *)
+ # Example: almalinux-8-amd64 -> almalinux
+ # Example: amazon-linux-2023-amd64 -> amazon-linux
+ distribution="${ID%-*-*}"
+ ;;
+ esac
+ echo "DISTRIBUTION=${distribution}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> amd64
+ architecture="${ID##*-}"
+ echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> almalinux-8
+ target="${ID%-*}"
+ case "${target}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: almalinux-8 -> almalinux-8-aarch64
+ target="${target}-aarch64"
+ fi
+ echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ *)
+ echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: ubuntu-noble -> ubuntu-noble-arm64
+ target="${target}-arm64"
+ fi
+ echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ esac
+ echo "TARGET=${target}" >> "${GITHUB_ENV}"
+
+ if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+ # Example: apache-arrow-21.0.0-rc0 -> 21.0.0-rc0
+ version="${GITHUB_REF_NAME#apache-arrow-}"
+ echo "ARROW_VERSION=${version}" >> "${GITHUB_ENV}"
+ fi
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: package-linux-${{ matrix.id }}-${{ hashFiles('cpp/**', 'c_glib/**') }}
+ restore-keys: package-linux-${{ matrix.id }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Prepare apache-arrow-apt-source for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-apt-source/apt
+ for target in *-*; do
+ cp -a ${target} ${target}-arm64
+ done
+ popd
+ - name: Prepare apache-arrow-release for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-release/yum
+ for target in *-*; do
+ cp -a ${target} ${target}-aarch64
+ done
+ popd
+ - name: Update version
+ if: github.ref_type != 'tag'
+ run: |
+ pushd dev/tasks/linux-packages
+ rake version:update
+ popd
+ - name: Login to GitHub Container registry
+ uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 # v3.5.0
+ with:
+ registry: ghcr.io
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+ - name: Wait for creating GitHub Release
+ if: github.ref_type == 'tag'
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ run: |
+ dev/release/utils-watch-gh-workflow.sh \
+ ${GITHUB_REF_NAME} \
+ release_candidate.yml
Review Comment:
Yes, that makes sense, I was trying to find the logic where this is decided but can't seem to find (on the ruby code) where are we deciding that there is a release candidate. I can see we are indeed downloading from the release on the fork tests so it was just to understand how the rake task flow works:
```
** Execute apt:build
/usr/bin/ruby3.2 -S rake apt:build
cd apache-arrow
cd /home/runner/work/arrow/arrow/dev/tasks/linux-packages/../../..
Downloading... https://github.com/kou/arrow/releases/download/apache-arrow-22.0.0-rc4/apache-arrow-22.0.0.tar.gz
```
So my question, not really related to the PR but for overall understanding of our ruby code, can you point me to where are we deciding on the ruby code that this is a release candidate and we should proceed to download?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #47600:
URL: https://github.com/apache/arrow/pull/47600#issuecomment-3333128599
Can I work on this next week? I'll push something to this branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #47600:
URL: https://github.com/apache/arrow/pull/47600#issuecomment-3333233136
> Can I work on this next week? I'll push something to this branch.
Absolutely. I will try to deeply review the changes for simplifying the repositories structure so I can learn more about how we build those packages. It would be good to spread the knowledge to help maintaining those.
Thank you very much @kou .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2372341942
##########
dev/release/04-binary-download.sh:
##########
@@ -39,3 +42,22 @@ crossbow_job_prefix="release-${version_with_rc}"
: ${CROSSBOW_JOB_ID:="${crossbow_job_prefix}-${CROSSBOW_JOB_NUMBER}"}
archery crossbow download-artifacts --no-fetch ${CROSSBOW_JOB_ID} "$@"
+
+# Wait for the GitHub Workflow that creates the Linux packages
+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
+
+RUN_ID=$(get_run_id)
+# Download the artifacts created by the package_linux.yml workflow
+download_artifacts "${SOURCE_DIR}/../../packages/${CROSSBOW_JOB_ID}"
Review Comment:
I want to upload artifacts to GitHub Release only when `apache-arrow-${VERSION}-rc${RC}` tag is pushed. Can we use `on.push.tags` instead of `workflow_dispatch` for RC? (I can provide changes for the approach.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2372243104
##########
dev/release/04-binary-download.sh:
##########
@@ -39,3 +42,22 @@ crossbow_job_prefix="release-${version_with_rc}"
: ${CROSSBOW_JOB_ID:="${crossbow_job_prefix}-${CROSSBOW_JOB_NUMBER}"}
archery crossbow download-artifacts --no-fetch ${CROSSBOW_JOB_ID} "$@"
+
+# Wait for the GitHub Workflow that creates the Linux packages
+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
+
+RUN_ID=$(get_run_id)
Review Comment:
Ah, you want to use a function defined in `utils-watch-gh-workflow.sh`.
##########
.github/workflows/package_linux.yml:
##########
@@ -0,0 +1,298 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Package Linux
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ tags:
+ - '**'
+ pull_request:
+ paths:
+ - '.dockerignore'
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ - 'testing'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+ workflow_dispatch:
+ inputs:
+ version:
+ description: "The Arrow version"
+ type: string
+ required: true
+ no_rc_version:
+ description: "The Arrow version without RC"
+ type: string
+ required: true
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ contents: read
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ parent-workflow: package_linux
+
+ package:
+ name: ${{ matrix.id }}
+ runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
+ needs: check-labels
+ if: >-
+ needs.check-labels.outputs.force == 'true' ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: Package: Linux')
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ id:
+ - almalinux-8-amd64
+ - almalinux-8-arm64
+ - almalinux-9-amd64
+ - almalinux-9-arm64
+ - almalinux-10-amd64
+ - almalinux-10-arm64
+ - amazon-linux-2023-amd64
+ - amazon-linux-2023-arm64
+ - centos-9-stream-amd64
+ - centos-9-stream-arm64
+ - centos-7-amd64
+ - debian-bookworm-amd64
+ - debian-bookworm-arm64
+ - debian-trixie-amd64
+ - debian-trixie-arm64
+ - debian-forky-amd64
+ - debian-forky-arm64
+ - ubuntu-jammy-amd64
+ - ubuntu-jammy-arm64
+ - ubuntu-noble-amd64
+ - ubuntu-noble-arm64
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ ARROW_VERSION: ${{ inputs.version || ''}}
+ NO_RC_VERSION: ${{ inputs.no_rc_version || ''}}
+ steps:
+ - name: Prepare environment variables
+ env:
+ ID: ${{ matrix.id }}
+ run: |
+ set -ex
+ # Example: almalinux-8-amd64 -> amd64
+ architecture="${ID##*-}"
+ echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> almalinux-8
+ target="${ID%-*}"
+ case "${target}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
+ echo "UPLOAD_EXTENSIONS=rpm" >> "${GITHUB_ENV}"
+ if [[ "${architecture}" == "arm64" ]]; then
+ # Example: almalinux-8 -> almalinux-8-aarch64
+ target="${target}-aarch64"
+ fi
+ echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ *)
+ echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
+ upload_extensions=(ddeb deb debian.tar.xz .dsc .orig.tar.gz)
+ echo "UPLOAD_EXTENSIONS=${upload_extensions[*]}" >> "${GITHUB_ENV}"
+ if [[ "${architecture}" == "arm64" ]]; then
+ # Example: ubuntu-noble -> ubuntu-noble-arm64
+ target="${target}-arm64"
+ fi
+ echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ esac
+ echo "TARGET=${target}" >> "${GITHUB_ENV}"
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ if: runner.os == 'Linux' && runner.arch == 'X64'
+ shell: bash
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: package-linux-${{ matrix.id }}-${{ hashFiles('cpp/**') }}
+ restore-keys: package-linux-${{ matrix.id }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Prepare apache-arrow-apt-source for arm64
+ if: ${{ env.ARCHITECTURE == 'arm64' }}
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-apt-source/apt
+ for target in *-*; do
+ cp -a ${target} ${target}-arm64
+ done
+ popd
+ - name: Prepare apache-arrow-release for arm64
+ if: ${{ env.ARCHITECTURE == 'arm64' }}
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-release/yum
+ for target in *-*; do
+ cp -a ${target} ${target}-aarch64
+ done
+ popd
+ - name: Get Arrow Version
+ id: get-arrow-version
+ run: |
+ # In case of dev (no workflow_dispatch inputs) compute ARROW_VERSION and
+ # use the same version for NO_RC_VERSION.
+ if [ -z "${ARROW_VERSION}" ]; then
+ ARROW_VERSION=$(git describe --tags --abbrev=0 --match "apache-arrow-[0-9]*.*" | sed 's/^apache-arrow-//')
+ ARROW_VERSION="${ARROW_VERSION}$(date +%Y%m%d)"
+ echo "ARROW_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ echo "NO_RC_VERSION=${ARROW_VERSION}" >> $GITHUB_ENV
+ fi
+ - name: Build
+ run: |
+ set -e
+ pushd dev/tasks/linux-packages
+ rake version:update ARROW_RELEASE_TIME="$(date --iso-8601=seconds)"
+ rake docker:pull || :
+ rake --trace ${TASK_NAMESPACE}:build BUILD_DIR=build
+ popd
+ env:
+ REPO: ghcr.io/${{ github.repository }}-package-linux
+ - name: Login to Dockerhub
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 # v3.5.0
+ with:
+ registry: ghcr.io
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN}}
+ - name: Docker Push
+ continue-on-error: true
+ if: >-
+ success() &&
+ github.event_name == 'push' &&
+ github.repository == 'apache/arrow' &&
+ github.ref_name == 'main'
+ shell: bash
+ run: |
+ pushd dev/tasks/linux-packages
+ rake docker:push
+ popd
+ env:
+ REPO: ${{ secrets.REPO }}
+ - name: Build artifact tarball
+ shell: bash
+ run: |
+ set -ex
+ pushd dev/tasks/linux-packages
+ tar cvzf ${{ matrix.id }}.tar.gz */${TASK_NAMESPACE}/repositories
Review Comment:
Let's simplify paths more to simplify `binary-task.rb`:
```suggestion
mkdir -p artifacts
cp -a */${TASK_NAMESPACE}/repositories/* artifacts/
tar cvzf ${{ matrix.id }}.tar.gz -C artifacts .
```
##########
dev/tasks/linux-packages/helper.rb:
##########
@@ -47,13 +47,13 @@ def arrow_source_dir
def detect_version(release_time)
version_env = ENV["ARROW_VERSION"]
- return version_env if version_env
+ return version_env unless version_env.nil? || version_env.empty?
Review Comment:
We can simplify this:
```suggestion
return version_env unless version_env.to_s.empty?
```
##########
dev/release/binary-task.rb:
##########
@@ -1645,45 +1645,37 @@ def define_apt_rc_tasks
progress_label = "Copying: #{distribution} #{code_name}"
progress_reporter = ProgressReporter.new(progress_label)
- distribution_dir = "#{incoming_dir}/#{distribution}"
- pool_dir = "#{distribution_dir}/pool/#{code_name}"
- rm_rf(pool_dir, verbose: verbose?)
- mkdir_p(pool_dir, verbose: verbose?)
- source_dir_prefix = "#{artifacts_dir}/#{distribution}-#{code_name}"
- # apache/arrow uses debian-bookworm-{amd64,arm64} but
- # apache/arrow-adbc uses debian-bookworm. So the following
- # glob must much both of them.
- Dir.glob("#{source_dir_prefix}*/*") do |path|
- base_name = File.basename(path)
+ destination_prefix = "#{incoming_dir}/#{distribution}"
+ rm_rf(destination_prefix, verbose: verbose?)
+
+ # Copy the entire repository structure
+ source_pattern = "#{artifacts_dir}/#{distribution}-#{code_name}*/*/" \
+ "apt/repositories/#{distribution}"
+ Dir.glob(source_pattern) do |repo_source|
+ progress_reporter.increment_max
+ mkdir_p(File.dirname(destination_prefix), verbose: verbose?)
+ cp_r(repo_source, destination_prefix, preserve: true, verbose: verbose?)
+ progress_reporter.advance
+ end
+
+ # Create latest package links after copying
+ Dir.glob("#{destination_prefix}/**/*-apt-source_*.deb") do |apt_source_path|
+ base_name = File.basename(apt_source_path)
package_name = ENV["DEB_PACKAGE_NAME"]
Review Comment:
We can remove this because `*-apt-source` is only used in this block. We don't need to customize it by an environment variable.
##########
dev/release/binary-task.rb:
##########
@@ -2043,59 +2035,23 @@ def define_yum_rc_tasks
progress_label = "Copying: #{distribution} #{distribution_version}"
progress_reporter = ProgressReporter.new(progress_label)
- destination_prefix = [
- incoming_dir,
- distribution,
- distribution_version,
- ].join("/")
+ destination_prefix = "#{incoming_dir}/#{distribution}/#{distribution_version}"
Review Comment:
Oh... This part uses `prefix` not `dir`... Sorry... This should be `dir`...
##########
dev/release/04-binary-download.sh:
##########
@@ -39,3 +42,22 @@ crossbow_job_prefix="release-${version_with_rc}"
: ${CROSSBOW_JOB_ID:="${crossbow_job_prefix}-${CROSSBOW_JOB_NUMBER}"}
archery crossbow download-artifacts --no-fetch ${CROSSBOW_JOB_ID} "$@"
+
+# Wait for the GitHub Workflow that creates the Linux packages
+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
+
+RUN_ID=$(get_run_id)
+# Download the artifacts created by the package_linux.yml workflow
+download_artifacts "${SOURCE_DIR}/../../packages/${CROSSBOW_JOB_ID}"
Review Comment:
How about using GitHub Release for RC instead of GitHub Actions artifacts? If we use GitHub Release, we don't need `RUN_ID`: `gh release download apache-arrow-${VERSION}-rc${RC} ...`
##########
dev/release/binary-task.rb:
##########
@@ -1645,45 +1645,37 @@ def define_apt_rc_tasks
progress_label = "Copying: #{distribution} #{code_name}"
progress_reporter = ProgressReporter.new(progress_label)
- distribution_dir = "#{incoming_dir}/#{distribution}"
- pool_dir = "#{distribution_dir}/pool/#{code_name}"
- rm_rf(pool_dir, verbose: verbose?)
- mkdir_p(pool_dir, verbose: verbose?)
- source_dir_prefix = "#{artifacts_dir}/#{distribution}-#{code_name}"
- # apache/arrow uses debian-bookworm-{amd64,arm64} but
- # apache/arrow-adbc uses debian-bookworm. So the following
- # glob must much both of them.
- Dir.glob("#{source_dir_prefix}*/*") do |path|
- base_name = File.basename(path)
+ destination_prefix = "#{incoming_dir}/#{distribution}"
Review Comment:
Thanks for writing Ruby!
Sure. I can help. Can I push something to this branch if it's needed?
BTW, `destination_dir` is better for this variable name because this is a directory name not a prefix of path. (In this code, "prefix" is used for `abc` for `abcde` directory and "dir" is used for `abcde` directory.)
##########
dev/release/utils-watch-gh-workflow.sh:
##########
@@ -30,17 +30,30 @@ TAG=$1
WORKFLOW=$2
: "${REPOSITORY:=${GITHUB_REPOSITORY:-apache/arrow}}"
+get_run_id() {
+ gh run list \
+ --branch "${TAG}" \
+ --jq '.[].databaseId' \
+ --json databaseId \
+ --limit 1 \
+ --repo "${REPOSITORY}" \
+ --workflow "${WORKFLOW}"
+}
+
+download_artifacts() {
+ RUN_ID=$(get_run_id)
+ echo "Downloading artitfacts for workflow with ID: ${RUN_ID}"
+ gh run download \
+ ${RUN_ID} \
+ --repo "${REPOSITORY}" \
+ --dir "$1"
+}
Review Comment:
Hmm. Defining this function in this file isn't good because this file name is `watch-gh-workflow`.
##########
dev/release/04-binary-download.sh:
##########
@@ -39,3 +42,22 @@ crossbow_job_prefix="release-${version_with_rc}"
: ${CROSSBOW_JOB_ID:="${crossbow_job_prefix}-${CROSSBOW_JOB_NUMBER}"}
archery crossbow download-artifacts --no-fetch ${CROSSBOW_JOB_ID} "$@"
+
+# Wait for the GitHub Workflow that creates the Linux packages
+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
Review Comment:
We don't need `.` here because all information is passed via arguments:
```suggestion
"${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
```
##########
dev/release/post-05-update-gh-release-notes.sh:
##########
@@ -36,7 +36,7 @@ WORKFLOW="release.yml"
# Wait for the GitHub Workflow that creates the GitHub Release
# to finish before updating the release notes.
-"${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${TAG}" "${WORKFLOW}"
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${TAG}" "${WORKFLOW}"
Review Comment:
We don't need `.` here.
##########
dev/release/binary-task.rb:
##########
@@ -1645,45 +1645,37 @@ def define_apt_rc_tasks
progress_label = "Copying: #{distribution} #{code_name}"
progress_reporter = ProgressReporter.new(progress_label)
- distribution_dir = "#{incoming_dir}/#{distribution}"
- pool_dir = "#{distribution_dir}/pool/#{code_name}"
- rm_rf(pool_dir, verbose: verbose?)
- mkdir_p(pool_dir, verbose: verbose?)
- source_dir_prefix = "#{artifacts_dir}/#{distribution}-#{code_name}"
- # apache/arrow uses debian-bookworm-{amd64,arm64} but
- # apache/arrow-adbc uses debian-bookworm. So the following
- # glob must much both of them.
Review Comment:
Yes.
But I can change ADBC to simplify this script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #47600:
URL: https://github.com/apache/arrow/pull/47600#issuecomment-3332920761
@kou I think there are two main things missing here:
- whether we want to further simplify the repositories structure and the `.tar.gz` (this can be done on a separate PR)
- whether we want to use the tag instead of workflow dispatch for the release and use `gh release download` instead of `gh run download`. If we don't want to do this on this PR I should probably rename `utils-watch-gh-workflow.sh` to `utils-gh-workflow.sh` so I can source the file and add the new function.
I can work on the second point regardless of the approach (unless you want to do the tag part).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #47600:
URL: https://github.com/apache/arrow/pull/47600#issuecomment-3355220912
(I'll work on this tomorrow...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47600:
URL: https://github.com/apache/arrow/pull/47600#issuecomment-3311455208
:warning: GitHub issue #47582 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2406610895
##########
.github/workflows/cpp_extra.yml:
##########
@@ -75,47 +79,11 @@ permissions:
jobs:
check-labels:
- name: Check labels
- runs-on: ubuntu-latest
- timeout-minutes: 5
- outputs:
- ci-extra: ${{ steps.check.outputs.ci-extra }}
- steps:
- - name: Checkout Arrow
- if: github.event_name == 'pull_request'
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
- - name: Check
- id: check
- env:
- GH_TOKEN: ${{ github.token }}
- run: |
- case "${GITHUB_EVENT_NAME}" in
- push|schedule)
- ci_extra=true
- ;;
- pull_request)
- n_ci_extra_labels=$(
- gh pr view ${{ github.event.number }} \
- --jq '.labels[].name | select(. == "CI: Extra")' \
- --json labels \
- --repo ${GITHUB_REPOSITORY} | wc -l)
- if [ "${n_ci_extra_labels}" -eq 1 ]; then
- ci_extra=true
- else
- git fetch origin ${GITHUB_BASE_REF}
- if git diff --stat origin/${GITHUB_BASE_REF}.. | \
- grep \
- --fixed-strings ".github/workflows/cpp_extra.yml" \
- --quiet; then
- ci_extra=true
- else
- ci_extra=false
- fi
- fi
- ;;
- esac
-
- echo "ci-extra=${ci_extra}" >> "${GITHUB_OUTPUT}"
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ label: "CI: Extra"
Review Comment:
I've created a couple of labels: `CI: Extra: Package: Linux` and `CI: Extra: C++`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #47600:
URL: https://github.com/apache/arrow/pull/47600#discussion_r2406664670
##########
.github/workflows/package_linux.yml:
##########
@@ -0,0 +1,313 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: Package Linux
+
+on:
+ push:
+ branches:
+ - '**'
+ - '!dependabot/**'
+ paths:
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ tags:
+ - "apache-arrow-*-rc*"
+ pull_request:
+ paths:
+ - '.github/workflows/check_labels.yml'
+ - '.github/workflows/package_linux.yml'
+ - '.github/workflows/report_ci.yml'
+ - 'cpp/**'
+ - 'c_glib/**'
+ - 'dev/tasks/linux-packages/**'
+ - 'format/Flight.proto'
+ types:
+ - labeled
+ - opened
+ - reopened
+ - synchronize
+ schedule:
+ - cron: "0 2 * * *"
+
+concurrency:
+ group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ cancel-in-progress: true
+
+permissions:
+ # Upload to GitHub Release
+ contents: write
+
+jobs:
+ check-labels:
+ uses: ./.github/workflows/check_labels.yml
+ secrets: inherit
+ with:
+ parent-workflow: package_linux
+
+ package:
+ name: ${{ matrix.id }}
+ runs-on: ${{ contains(matrix.id, 'amd64') && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
+ needs: check-labels
+ if: >-
+ needs.check-labels.outputs.force == 'true' ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra') ||
+ contains(fromJSON(needs.check-labels.outputs.ci-extra-labels || '[]'), 'CI: Extra: Package: Linux')
+ timeout-minutes: 75
+ strategy:
+ fail-fast: false
+ matrix:
+ id:
+ - almalinux-8-amd64
+ - almalinux-8-arm64
+ - almalinux-9-amd64
+ - almalinux-9-arm64
+ - almalinux-10-amd64
+ - almalinux-10-arm64
+ - amazon-linux-2023-amd64
+ - amazon-linux-2023-arm64
+ - centos-9-stream-amd64
+ - centos-9-stream-arm64
+ - centos-7-amd64
+ - debian-bookworm-amd64
+ - debian-bookworm-arm64
+ - debian-trixie-amd64
+ - debian-trixie-arm64
+ - debian-forky-amd64
+ - debian-forky-arm64
+ - ubuntu-jammy-amd64
+ - ubuntu-jammy-arm64
+ - ubuntu-noble-amd64
+ - ubuntu-noble-arm64
+ env:
+ DOCKER_VOLUME_PREFIX: ".docker/"
+ steps:
+ - name: Checkout Arrow
+ uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: Free up disk space
+ run: |
+ ci/scripts/util_free_space.sh
+ - name: Prepare environment variables
+ env:
+ ID: ${{ matrix.id }}
+ run: |
+ set -x
+ case "${ID}" in
+ centos-*)
+ # Example: centos-9-stream-amd64 -> centos
+ distribution="${ID%%-*}"
+ ;;
+ *)
+ # Example: almalinux-8-amd64 -> almalinux
+ # Example: amazon-linux-2023-amd64 -> amazon-linux
+ distribution="${ID%-*-*}"
+ ;;
+ esac
+ echo "DISTRIBUTION=${distribution}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> amd64
+ architecture="${ID##*-}"
+ echo "ARCHITECTURE=${architecture}" >> "${GITHUB_ENV}"
+ # Example: almalinux-8-amd64 -> almalinux-8
+ target="${ID%-*}"
+ case "${target}" in
+ almalinux-*|amazon-linux-*|centos-*)
+ echo "TASK_NAMESPACE=yum" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: almalinux-8 -> almalinux-8-aarch64
+ target="${target}-aarch64"
+ fi
+ echo "YUM_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ *)
+ echo "TASK_NAMESPACE=apt" >> "${GITHUB_ENV}"
+ if [ "${architecture}" = "arm64" ]; then
+ # Example: ubuntu-noble -> ubuntu-noble-arm64
+ target="${target}-arm64"
+ fi
+ echo "APT_TARGETS=${target}" >> "${GITHUB_ENV}"
+ ;;
+ esac
+ echo "TARGET=${target}" >> "${GITHUB_ENV}"
+
+ if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+ # Example: apache-arrow-21.0.0-rc0 -> 21.0.0-rc0
+ version="${GITHUB_REF_NAME#apache-arrow-}"
+ echo "ARROW_VERSION=${version}" >> "${GITHUB_ENV}"
+ fi
+ - name: Cache Docker Volumes
+ uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+ with:
+ path: .docker
+ key: package-linux-${{ matrix.id }}-${{ hashFiles('cpp/**', 'c_glib/**') }}
+ restore-keys: package-linux-${{ matrix.id }}-
+ - name: Set up Ruby
+ run: |
+ sudo apt update
+ sudo apt install -y \
+ rake \
+ ruby \
+ ruby-dev
+ - name: Prepare apache-arrow-apt-source for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-apt-source/apt
+ for target in *-*; do
+ cp -a ${target} ${target}-arm64
+ done
+ popd
+ - name: Prepare apache-arrow-release for arm64
+ if: env.ARCHITECTURE == 'arm64'
+ run: |
+ pushd dev/tasks/linux-packages/apache-arrow-release/yum
+ for target in *-*; do
+ cp -a ${target} ${target}-aarch64
+ done
+ popd
+ - name: Update version
+ if: github.ref_type != 'tag'
+ run: |
+ pushd dev/tasks/linux-packages
+ rake version:update
+ popd
+ - name: Login to GitHub Container registry
+ uses: docker/login-action@184bdaa0721073962dff0199f1fb9940f07167d1 # v3.5.0
+ with:
+ registry: ghcr.io
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+ - name: Wait for creating GitHub Release
+ if: github.ref_type == 'tag'
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ run: |
+ dev/release/utils-watch-gh-workflow.sh \
+ ${GITHUB_REF_NAME} \
+ release_candidate.yml
Review Comment:
This is waiting for the GitHub release to be generated in order to upload the artifacts there, right? Do we have to wait for that to be finished before building for any reason or could we just do it after we build before uploading the artifacts? I am thinking on not having to be idle here for those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org