You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/08/11 07:38:02 UTC
[GitHub] [pulsar] BewareMyPower opened a new pull request, #17064: [fix][C++ client] Fix rpm and deb packaging
BewareMyPower opened a new pull request, #17064:
URL: https://github.com/apache/pulsar/pull/17064
### Motivation
For branch-2.11 and master branches, the rpm and deb packaging is
broken.
> /usr/bin/env: 'python3': No such file or directory
It's caused by [PIP-155](https://github.com/apache/pulsar/pull/15376),
which removes the Python2 support. In addition, the Dockerfiles of `rpm`
and `deb` subdirectories of `pulsar-client-cpp/pkg` don't install Boost
for Python3 well. But the rpm build disables the
`BUILD_PYTHON_WRAPPER` option while the deb build doesn't.
### Modifications
- Fix Dockerfiles under `pkg` and `rpm` subdirectories and speed up the
`docker build` process:
- Install python3 to fix the bug
- Download the CMake 3.24.0 binaries directly instead of building
from source
- Upgrade Boost to 1.79 so that only headers are required (the
boost::regex module is required for CentOS 7 because the default
GCC is 4.8 that there is something wrong with std::regex)
- Add `-j8` option when compiling some large dependencies that uses 8
threads to speed the compilation
- Remove the `-v` option of `tar` to avoid showing too many info
- Check the `BUILD_IMAGE` environment variable in `docker-build-xxx.sh`
under `rpm` and `deb` subdirectories. If it's defined, build the image
rather than pull the image from DockerHub before executing the build
script. Because currently only a few committers have the permission to
push the Docker images.
- Add two workflows to build RPM and DEB packages if the files under
`pulsar-client-cpp` changed. They build the docker images and run the
build script for RPM and DEB packaging.
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [x] `doc-not-needed`
I will update the docs in Wiki pages. The dos in the repository don't need update.
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #17064: [fix][C++ client] Fix rpm and deb packaging
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #17064:
URL: https://github.com/apache/pulsar/pull/17064#issuecomment-1211687772
@BewareMyPower Please provide a correct documentation label for your PR.
Instructions see [Pulsar Documentation Label Guide](https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #17064: [fix][C++ client] Fix rpm and deb packaging
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #17064:
URL: https://github.com/apache/pulsar/pull/17064#issuecomment-1211941623
@BewareMyPower Please provide a correct documentation label for your PR.
Instructions see [Pulsar Documentation Label Guide](https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on a diff in pull request #17064: [fix][C++ client] Fix rpm and deb packaging
Posted by GitBox <gi...@apache.org>.
shibd commented on code in PR #17064:
URL: https://github.com/apache/pulsar/pull/17064#discussion_r945676921
##########
.github/workflows/ci-cpp-deb-pkg.yaml:
##########
@@ -0,0 +1,97 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: CI - CPP DEB packaging
+on:
+ pull_request:
+ branches:
+ - master
+ paths:
+ - '.github/workflows/**'
+ - 'pulsar-client-cpp/**'
+ push:
+ branches:
+ - branch-*
+ paths:
+ - '.github/workflows/**'
+ - 'pulsar-client-cpp/**'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+
+ cpp-deb-packaging:
+ name:
+ runs-on: ubuntu-latest
+ timeout-minutes: 120
+
+ steps:
+ - name: checkout
+ uses: actions/checkout@v2
+
+ - name: Tune Runner VM
+ uses: ./.github/actions/tune-runner-vm
+
+ - name: Detect changed files
+ id: changes
+ uses: apache/pulsar-test-infra/paths-filter@master
+ with:
+ filters: .github/changes-filter.yaml
+ list-files: csv
+
+ - name: Check changed files
+ id: check_changes
+ run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}"
+
+ - name: Cache local Maven repository
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ uses: actions/cache@v2
+ with:
+ path: |
+ ~/.m2/repository/*/*/*
+ !~/.m2/repository/org/apache/pulsar
+ key: ${{ runner.os }}-m2-dependencies-core-modules-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ ${{ runner.os }}-m2-dependencies-core-modules-
+
+ - name: Set up JDK 17
+ uses: actions/setup-java@v2
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ with:
+ distribution: 'temurin'
+ java-version: 17
+
+ - name: clean disk
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ run: |
+ sudo apt clean
+ docker rmi $(docker images -q) -f
+ df -h
+
+ - name: Package Pulsar source
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ run: mvn -B -ntp -q clean package -pl pulsar-client-api -am -DskipTests
+
+ - name: Build Debian packages
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ run: |
+ echo "Build Debian packages"
+ BUILD_IMAGE=1 pulsar-client-cpp/pkg/deb/docker-build-deb.sh
Review Comment:
Ok, Thanks for your reply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #17064: [fix][C++ client] Fix rpm and deb packaging
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on PR #17064:
URL: https://github.com/apache/pulsar/pull/17064#issuecomment-1211939834
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #17064: [fix][C++ client] Fix rpm and deb packaging
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on PR #17064:
URL: https://github.com/apache/pulsar/pull/17064#issuecomment-1215133786
Also worked in my M1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #17064: [fix][C++ client] Fix rpm and deb packaging
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #17064:
URL: https://github.com/apache/pulsar/pull/17064#issuecomment-1211702239
@BewareMyPower Please provide a correct documentation label for your PR.
Instructions see [Pulsar Documentation Label Guide](https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #17064: [fix][C++ client] Fix rpm and deb packaging
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on PR #17064:
URL: https://github.com/apache/pulsar/pull/17064#issuecomment-1211650124
I add the `release/blocker` label first. @Technoboy- I think the [Step 3 in release process](https://github.com/apache/pulsar/wiki/Release-process#31-build-rpm-and-deb-packages) should have failed when you released 2.11.0. Can you confirm it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on a diff in pull request #17064: [fix][C++ client] Fix rpm and deb packaging
Posted by GitBox <gi...@apache.org>.
shibd commented on code in PR #17064:
URL: https://github.com/apache/pulsar/pull/17064#discussion_r945592395
##########
.github/workflows/ci-cpp-deb-pkg.yaml:
##########
@@ -0,0 +1,97 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: CI - CPP DEB packaging
+on:
+ pull_request:
+ branches:
+ - master
+ paths:
+ - '.github/workflows/**'
+ - 'pulsar-client-cpp/**'
+ push:
+ branches:
+ - branch-*
+ paths:
+ - '.github/workflows/**'
+ - 'pulsar-client-cpp/**'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+
+ cpp-deb-packaging:
+ name:
+ runs-on: ubuntu-latest
+ timeout-minutes: 120
+
+ steps:
+ - name: checkout
+ uses: actions/checkout@v2
+
+ - name: Tune Runner VM
+ uses: ./.github/actions/tune-runner-vm
+
+ - name: Detect changed files
+ id: changes
+ uses: apache/pulsar-test-infra/paths-filter@master
+ with:
+ filters: .github/changes-filter.yaml
+ list-files: csv
+
+ - name: Check changed files
+ id: check_changes
+ run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}"
+
+ - name: Cache local Maven repository
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ uses: actions/cache@v2
+ with:
+ path: |
+ ~/.m2/repository/*/*/*
+ !~/.m2/repository/org/apache/pulsar
+ key: ${{ runner.os }}-m2-dependencies-core-modules-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ ${{ runner.os }}-m2-dependencies-core-modules-
+
+ - name: Set up JDK 17
+ uses: actions/setup-java@v2
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ with:
+ distribution: 'temurin'
+ java-version: 17
+
+ - name: clean disk
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ run: |
+ sudo apt clean
+ docker rmi $(docker images -q) -f
+ df -h
+
+ - name: Package Pulsar source
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ run: mvn -B -ntp -q clean package -pl pulsar-client-api -am -DskipTests
+
+ - name: Build Debian packages
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ run: |
+ echo "Build Debian packages"
+ BUILD_IMAGE=1 pulsar-client-cpp/pkg/deb/docker-build-deb.sh
Review Comment:
Can we upload the built image to the docker hub? This reduces build time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on a diff in pull request #17064: [fix][C++ client] Fix rpm and deb packaging
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on code in PR #17064:
URL: https://github.com/apache/pulsar/pull/17064#discussion_r945618405
##########
.github/workflows/ci-cpp-deb-pkg.yaml:
##########
@@ -0,0 +1,97 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: CI - CPP DEB packaging
+on:
+ pull_request:
+ branches:
+ - master
+ paths:
+ - '.github/workflows/**'
+ - 'pulsar-client-cpp/**'
+ push:
+ branches:
+ - branch-*
+ paths:
+ - '.github/workflows/**'
+ - 'pulsar-client-cpp/**'
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+
+jobs:
+
+ cpp-deb-packaging:
+ name:
+ runs-on: ubuntu-latest
+ timeout-minutes: 120
+
+ steps:
+ - name: checkout
+ uses: actions/checkout@v2
+
+ - name: Tune Runner VM
+ uses: ./.github/actions/tune-runner-vm
+
+ - name: Detect changed files
+ id: changes
+ uses: apache/pulsar-test-infra/paths-filter@master
+ with:
+ filters: .github/changes-filter.yaml
+ list-files: csv
+
+ - name: Check changed files
+ id: check_changes
+ run: echo "::set-output name=docs_only::${{ fromJSON(steps.changes.outputs.all_count) == fromJSON(steps.changes.outputs.docs_count) && fromJSON(steps.changes.outputs.docs_count) > 0 }}"
+
+ - name: Cache local Maven repository
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ uses: actions/cache@v2
+ with:
+ path: |
+ ~/.m2/repository/*/*/*
+ !~/.m2/repository/org/apache/pulsar
+ key: ${{ runner.os }}-m2-dependencies-core-modules-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ ${{ runner.os }}-m2-dependencies-core-modules-
+
+ - name: Set up JDK 17
+ uses: actions/setup-java@v2
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ with:
+ distribution: 'temurin'
+ java-version: 17
+
+ - name: clean disk
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ run: |
+ sudo apt clean
+ docker rmi $(docker images -q) -f
+ df -h
+
+ - name: Package Pulsar source
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ run: mvn -B -ntp -q clean package -pl pulsar-client-api -am -DskipTests
+
+ - name: Build Debian packages
+ if: ${{ steps.check_changes.outputs.docs_only != 'true' }}
+ run: |
+ echo "Build Debian packages"
+ BUILD_IMAGE=1 pulsar-client-cpp/pkg/deb/docker-build-deb.sh
Review Comment:
On one hand, see my PR description:
> Because currently only a few committers have the permission to
push the Docker images.
On the other hand, if these Dockerfiles (`pkg/deb/Dockerfile` and `pkg/rpm/Dockerfile`) changed, the changes can only be verified after uploading the images first. It's a manual operation and we cannot guarantee the docker image was built well. For example, if the docker image was built in an ARM64 architecture computer, it might not work well in the CI workflow.
Actually you can see it doesn't spend too much time building these images. I have already speeded up the build time for these Dockerfiles in the PR. In future, we can make other optimizations. If we use these images directly from Docker Hub, verifying the changes would be complicated and require some human operations.
<img width="842" alt="image" src="https://user-images.githubusercontent.com/18204803/184623301-fcb5726c-47a7-4558-bd1f-4646bba763d8.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- merged pull request #17064: [fix][C++ client] Fix rpm and deb packaging
Posted by GitBox <gi...@apache.org>.
Technoboy- merged PR #17064:
URL: https://github.com/apache/pulsar/pull/17064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org