You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by lh...@apache.org on 2022/09/13 14:19:16 UTC
[pulsar] branch master updated: [fix][cpp] Fix libcurl build failure when building deb package (#17614)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 0754ea105ea [fix][cpp] Fix libcurl build failure when building deb package (#17614)
0754ea105ea is described below
commit 0754ea105eab8c356ba18b2f4862cb1de1f6f1b6
Author: Yunze Xu <xy...@163.com>
AuthorDate: Tue Sep 13 22:18:56 2022 +0800
[fix][cpp] Fix libcurl build failure when building deb package (#17614)
* [fix][cpp] Fix libcurl build failure when building deb package
### Motivation
See https://github.com/apache/pulsar/pull/17538#issuecomment-1244898721
The root cause is when libcurl is built from source, it uses
[`ld`](https://linux.die.net/man/1/ld) to check if the `libcurl.so`
links to the correct dependencies in runtime. In Linux, a dynamic
library links to the paths of `/etc/ld.so.conf` by default. However,
different from other images like `centos:7` and `alpine`, this file
includes `/usr/lib/x86_64-linux-gnu` in `debian:9`.
```bash
$ cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf
$ cat /etc/ld.so.conf.d/*.conf
/usr/lib/x86_64-linux-gnu/libfakeroot
# libc default configuration
/usr/local/lib
# Multiarch support
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu
```
When libcurl is compiled, it links to the install path of libopenssl via
the `--with-ssl` option:
https://github.com/apache/pulsar/blob/1f50366768e76f1a5f7084f7972167f989ddd0af/pulsar-client-cpp/pkg/deb/Dockerfile#L85
i.e. `/usr/local/ssl/lib/libopenssl.so`.
However, after the `libcurl.so` is built, it links to
`/usr/lib/x86_64-linux-gnu/libssl.so.1.1`, see the following output:
```bash
$ ldd /usr/local/lib/libcurl.so
/usr/local/lib/libcurl.so: /usr/lib/x86_64-linux-gnu/libssl.so.1.1: version `OPENSSL_1_1_1' not found (required by /usr/local/lib/libcurl.so)
```
In `debian:9`, the default libopenssl version is 1.1.0:
```bash
$ strings /usr/lib/x86_64-linux-gnu/libssl.so.1.1 | grep OpenSSL
OpenSSL 1.1.0l 10 Sep 2019
```
The ABI compatibility is not guaranteed between 1.1.0l and 1.1.1n, see
https://abi-laboratory.pro/index.php?view=timeline&l=openssl.
### Modifications
Set the `LD_LIBRARY_PATH` to `/usr/local/ssl/lib` in the Dockerfile to
build deb package.
Actually it's not required for other images like `centos:7`, but it's
also good to add the `LD_LIBRARY_PATH` to them. So this PR set the
environment variable to them as well.
* Fix workflow so that cpp-tests isn't skipped
* Revisit workflow fix to cover doc only workflows too
* Fix multi-line condition
Co-authored-by: Lari Hotari <lh...@apache.org>
---
.github/workflows/pulsar-ci.yaml | 38 +++++++++++++++++++---
pulsar-client-cpp/docker/alpine/Dockerfile | 2 ++
.../docker/alpine/Dockerfile-alpine-3.8 | 2 ++
pulsar-client-cpp/docker/manylinux1/Dockerfile | 2 ++
pulsar-client-cpp/docker/manylinux2014/Dockerfile | 2 ++
pulsar-client-cpp/docker/manylinux_musl/Dockerfile | 2 ++
pulsar-client-cpp/pkg/deb/Dockerfile | 2 ++
pulsar-client-cpp/pkg/rpm/Dockerfile | 2 ++
8 files changed, 47 insertions(+), 5 deletions(-)
diff --git a/.github/workflows/pulsar-ci.yaml b/.github/workflows/pulsar-ci.yaml
index 51f98aee390..97b6bbeb025 100644
--- a/.github/workflows/pulsar-ci.yaml
+++ b/.github/workflows/pulsar-ci.yaml
@@ -476,15 +476,31 @@ jobs:
'changed_files_job',
'integration-tests'
]
- if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
+ if: always()
steps:
+ - name: check condition
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
+ run: |
+ if [[ ! ( ( \
+ "${{needs.changed_files_job.outputs.cpp_only}}" == "false" \
+ && "${{ needs.integration-tests.result }}" == "success" \
+ ) || ( \
+ "${{needs.changed_files_job.outputs.cpp_only}}" == "true" \
+ ) ) ]]; then
+ echo "Required jobs haven't been completed successfully."
+ exit 1
+ fi
+
- name: checkout
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
uses: actions/checkout@v2
- name: Tune Runner VM
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
uses: ./.github/actions/tune-runner-vm
- name: Cache local Maven repository
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
uses: actions/cache@v2
with:
path: |
@@ -495,18 +511,22 @@ jobs:
${{ runner.os }}-m2-dependencies-core-modules-
- name: Set up JDK 17
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
uses: actions/setup-java@v2
with:
distribution: 'temurin'
java-version: 17
- name: Clean Disk
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
uses: ./.github/actions/clean-disk
- name: Install gh-actions-artifact-client.js
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
uses: apache/pulsar-test-infra/gh-actions-artifact-client/dist@master
- name: Restore maven build results from Github artifact cache
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
run: |
cd $HOME
$GITHUB_WORKSPACE/build/pulsar_ci_tool.sh restore_tar_from_github_actions_artifacts pulsar-maven-repository-binaries
@@ -514,20 +534,23 @@ jobs:
$GITHUB_WORKSPACE/build/pulsar_ci_tool.sh restore_tar_from_github_actions_artifacts pulsar-server-distribution
- name: copy python tests
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
run: |
mvn -B -Pskip-all -ntp -pl pulsar-functions/instance package
- name: build cpp artifacts
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
run: |
echo "Build C++ client library"
pulsar-client-cpp/docker-build.sh
- name: run c++ tests
+ if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
run: pulsar-client-cpp/docker-tests.sh
- name: Upload test-logs
+ if: ${{ failure() && needs.changed_files_job.outputs.docs_only != 'true' }}
uses: actions/upload-artifact@v3
- if: failure()
continue-on-error: true
with:
name: cpp-tests-logs
@@ -1008,13 +1031,18 @@ jobs:
- name: Check that all required jobs were completed successfully
if: ${{ needs.changed_files_job.outputs.docs_only != 'true' }}
run: |
- if [[ ! ( \
- "${{ needs.unit-tests.result }}" == "success" \
+ if [[ ! ( ( \
+ "${{needs.changed_files_job.outputs.cpp_only}}" == "false" \
+ && "${{ needs.unit-tests.result }}" == "success" \
&& "${{ needs.integration-tests.result }}" == "success" \
&& "${{ needs.system-tests.result }}" == "success" \
&& "${{ needs.macos-build.result }}" == "success" \
&& "${{ needs.cpp-tests.result }}" == "success" \
- ) ]]; then
+ ) || ( \
+ "${{needs.changed_files_job.outputs.cpp_only}}" == "true" \
+ && "${{ needs.system-tests.result }}" == "success" \
+ && "${{ needs.cpp-tests.result }}" == "success" \
+ ) ) ]]; then
echo "Required jobs haven't been completed successfully."
exit 1
fi
diff --git a/pulsar-client-cpp/docker/alpine/Dockerfile b/pulsar-client-cpp/docker/alpine/Dockerfile
index d535a8ecf0e..c1c2974b56b 100644
--- a/pulsar-client-cpp/docker/alpine/Dockerfile
+++ b/pulsar-client-cpp/docker/alpine/Dockerfile
@@ -58,6 +58,8 @@ RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_1n.tar.gz
make -j8 && make install && \
rm -rf /OpenSSL_1_1_1n.tar.gz /openssl-OpenSSL_1_1_1n
+ENV LD_LIBRARY_PATH /usr/local/ssl/lib/:
+
# Download and copile protoubf
RUN curl -O -L https://github.com/google/protobuf/releases/download/v3.20.0/protobuf-cpp-3.20.0.tar.gz && \
tar xvfz protobuf-cpp-3.20.0.tar.gz && \
diff --git a/pulsar-client-cpp/docker/alpine/Dockerfile-alpine-3.8 b/pulsar-client-cpp/docker/alpine/Dockerfile-alpine-3.8
index 5b54eff8e8e..fb5cc08bf4a 100644
--- a/pulsar-client-cpp/docker/alpine/Dockerfile-alpine-3.8
+++ b/pulsar-client-cpp/docker/alpine/Dockerfile-alpine-3.8
@@ -58,6 +58,8 @@ RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_1n.tar.gz
make -j8 && make install && \
rm -rf /OpenSSL_1_1_1n.tar.gz /openssl-OpenSSL_1_1_1n
+ENV LD_LIBRARY_PATH /usr/local/ssl/lib/:
+
# Download and copile protoubf
RUN curl -O -L https://github.com/google/protobuf/releases/download/v3.20.0/protobuf-cpp-3.20.0.tar.gz && \
tar xvfz protobuf-cpp-3.20.0.tar.gz && \
diff --git a/pulsar-client-cpp/docker/manylinux1/Dockerfile b/pulsar-client-cpp/docker/manylinux1/Dockerfile
index ff9bd999705..9fef05aaeba 100644
--- a/pulsar-client-cpp/docker/manylinux1/Dockerfile
+++ b/pulsar-client-cpp/docker/manylinux1/Dockerfile
@@ -62,6 +62,8 @@ RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_1n.tar.gz
make && make install && \
rm -rf /OpenSSL_1_1_1n.tar.gz /openssl-OpenSSL_1_1_1n
+ENV LD_LIBRARY_PATH /usr/local/ssl/lib/:
+
# Download and compile boost
RUN curl -O -L https://boostorg.jfrog.io/artifactory/main/release/1.68.0/source/boost_1_68_0.tar.gz && \
tar xvfz boost_1_68_0.tar.gz && \
diff --git a/pulsar-client-cpp/docker/manylinux2014/Dockerfile b/pulsar-client-cpp/docker/manylinux2014/Dockerfile
index ce4e192c79b..5cab10cdc7d 100644
--- a/pulsar-client-cpp/docker/manylinux2014/Dockerfile
+++ b/pulsar-client-cpp/docker/manylinux2014/Dockerfile
@@ -63,6 +63,8 @@ RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_1n.tar.gz
make -j8 && make install && \
rm -rf /OpenSSL_1_1_1n.tar.gz /openssl-OpenSSL_1_1_1n
+ENV LD_LIBRARY_PATH /usr/local/ssl/lib/:
+
# Download and compile boost
RUN curl -O -L https://boostorg.jfrog.io/artifactory/main/release/1.78.0/source/boost_1_78_0.tar.gz && \
tar xvfz boost_1_78_0.tar.gz && \
diff --git a/pulsar-client-cpp/docker/manylinux_musl/Dockerfile b/pulsar-client-cpp/docker/manylinux_musl/Dockerfile
index 2c9ec058976..8eec24981c4 100644
--- a/pulsar-client-cpp/docker/manylinux_musl/Dockerfile
+++ b/pulsar-client-cpp/docker/manylinux_musl/Dockerfile
@@ -63,6 +63,8 @@ RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_1n.tar.gz
make -j8 && make install && \
rm -rf /OpenSSL_1_1_1n.tar.gz /openssl-OpenSSL_1_1_1n
+ENV LD_LIBRARY_PATH /usr/local/ssl/lib/:
+
# Download and compile boost
RUN curl -O -L https://boostorg.jfrog.io/artifactory/main/release/1.78.0/source/boost_1_78_0.tar.gz && \
tar xvfz boost_1_78_0.tar.gz && \
diff --git a/pulsar-client-cpp/pkg/deb/Dockerfile b/pulsar-client-cpp/pkg/deb/Dockerfile
index 9408d192e57..171c8292661 100644
--- a/pulsar-client-cpp/pkg/deb/Dockerfile
+++ b/pulsar-client-cpp/pkg/deb/Dockerfile
@@ -78,6 +78,8 @@ RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_1n.tar.gz
make -j8 && make install && \
rm -rf /OpenSSL_1_1_1n.tar.gz /openssl-OpenSSL_1_1_1n
+ENV LD_LIBRARY_PATH /usr/local/ssl/lib/:
+
# LibCurl
RUN curl -O -L https://github.com/curl/curl/releases/download/curl-7_61_0/curl-7.61.0.tar.gz && \
tar xfz curl-7.61.0.tar.gz && \
diff --git a/pulsar-client-cpp/pkg/rpm/Dockerfile b/pulsar-client-cpp/pkg/rpm/Dockerfile
index 3cdf0435206..9e4a057003e 100644
--- a/pulsar-client-cpp/pkg/rpm/Dockerfile
+++ b/pulsar-client-cpp/pkg/rpm/Dockerfile
@@ -79,6 +79,8 @@ RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_1n.tar.gz
make -j8 && make install && \
rm -rf /OpenSSL_1_1_1n.tar.gz /openssl-OpenSSL_1_1_1n
+ENV LD_LIBRARY_PATH /usr/local/ssl/lib/:
+
# LibCurl
RUN curl -O -L https://github.com/curl/curl/releases/download/curl-7_61_0/curl-7.61.0.tar.gz && \
tar xfz curl-7.61.0.tar.gz && \