You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/10/06 13:53:30 UTC
[GitHub] [pulsar-client-cpp] merlimat opened a new pull request, #19: Build images with deps versions
merlimat opened a new pull request, #19:
URL: https://github.com/apache/pulsar-client-cpp/pull/19
### Motivation
Since the dependencies are getting statically linked into the binary artifacts that we release, we need to have a good/easy way to upgrade the versions of such dependencies and make sure they are consistent for all the different builds.
### Modifications
1. Use a single `dependencies.yaml` file to declare the dependencies
2. The versions are then converted to variable inside the Dockerfiles
3. Create (and cache) the Docker images used for building in the Github actions job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] merlimat commented on a diff in pull request #19: Build images with deps versions
Posted by GitBox <gi...@apache.org>.
merlimat commented on code in PR #19:
URL: https://github.com/apache/pulsar-client-cpp/pull/19#discussion_r990146354
##########
.github/workflows/ci-build-binary-artifacts.yaml:
##########
@@ -43,8 +41,20 @@ jobs:
- name: Package Pulsar source
run: build-support/generate-source-archive.sh
+ - uses: docker/setup-buildx-action@v2
+ - run: build-support/copy-deps-versionfile.sh
+
+ - name: Build dependencies Docker image
+ uses: docker/build-push-action@v3
+ with:
+ context: ./pkg/deb
+ tags: ${{ env.TEST_TAG }}
Review Comment:
That’s a good point and a good catch. I thought it was auto assigned some unique tag, though it just ends up being an empty tag, so it goes using the default image, which is already on dockerhub. I’ll fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] BewareMyPower commented on a diff in pull request #19: Build images with deps versions
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on code in PR #19:
URL: https://github.com/apache/pulsar-client-cpp/pull/19#discussion_r989856809
##########
.github/workflows/ci-build-binary-artifacts.yaml:
##########
@@ -43,8 +41,20 @@ jobs:
- name: Package Pulsar source
run: build-support/generate-source-archive.sh
+ - uses: docker/setup-buildx-action@v2
+ - run: build-support/copy-deps-versionfile.sh
+
+ - name: Build dependencies Docker image
+ uses: docker/build-push-action@v3
+ with:
+ context: ./pkg/deb
+ tags: ${{ env.TEST_TAG }}
Review Comment:
Could you explain where the `TEST_TAG` environment variable is set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] BewareMyPower commented on a diff in pull request #19: Build images with deps versions
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on code in PR #19:
URL: https://github.com/apache/pulsar-client-cpp/pull/19#discussion_r989905922
##########
pkg/apk/Dockerfile:
##########
@@ -0,0 +1,112 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+FROM alpine:3.16
+
+ARG PLATFORM
+
+# perl is required to install OpenSSL
+RUN apk add \
+ build-base \
+ linux-headers \
+ abuild \
+ bash \
+ curl \
+ g++ \
+ make \
+ cmake \
+ python3 \
+ py3-pip \
+ perl \
+ sudo
+
+RUN pip3 install pyyaml
+
+ADD .build/dependencies.yaml /
+ADD .build/dep-version.py /usr/local/bin
+
+# Download and compile boost
+RUN BOOST_VERSION=$(dep-version.py boost) && \
+ BOOST_VERSION_UNDESRSCORE=$(echo $BOOST_VERSION | sed 's/\./_/g') && \
+ curl -O -L https://boostorg.jfrog.io/artifactory/main/release/${BOOST_VERSION}/source/boost_${BOOST_VERSION_UNDESRSCORE}.tar.gz && \
+ tar xfz boost_${BOOST_VERSION_UNDESRSCORE}.tar.gz && \
+ cd boost_${BOOST_VERSION_UNDESRSCORE} && \
+ ./bootstrap.sh --with-libraries=regex && \
+ ./b2 address-model=64 cxxflags=-fPIC link=static threading=multi variant=release install && \
+ rm -rf /boost_${BOOST_VERSION_UNDESRSCORE}.tar.gz /boost_${BOOST_VERSION_UNDESRSCORE}
+
+# Download and compile protobuf
+RUN PROTOBUF_VERSION=$(dep-version.py protobuf) && \
+ curl -O -L https://github.com/google/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-cpp-${PROTOBUF_VERSION}.tar.gz && \
+ tar xfz protobuf-cpp-${PROTOBUF_VERSION}.tar.gz && \
+ cd protobuf-${PROTOBUF_VERSION}/ && \
+ CXXFLAGS=-fPIC ./configure && \
+ make -j8 && make install && \
+ rm -rf /protobuf-cpp-${PROTOBUF_VERSION}.tar.gz /protobuf-${PROTOBUF_VERSION}
+
+# ZLib
+RUN ZLIB_VERSION=$(dep-version.py zlib) && \
+ curl -O -L https://github.com/madler/zlib/archive/v${ZLIB_VERSION}.tar.gz && \
+ tar xfz v${ZLIB_VERSION}.tar.gz && \
+ cd zlib-${ZLIB_VERSION} && \
+ CFLAGS="-fPIC -O3" ./configure && \
+ make -j8 && make install && \
+ rm -rf /v${ZLIB_VERSION}.tar.gz /zlib-${ZLIB_VERSION}
+
+# Zstandard
+RUN ZSTD_VERSION=$(dep-version.py zstd) && \
+ curl -O -L https://github.com/facebook/zstd/releases/download/v${ZSTD_VERSION}/zstd-${ZSTD_VERSION}.tar.gz && \
+ tar xfz zstd-${ZSTD_VERSION}.tar.gz && \
+ cd zstd-${ZSTD_VERSION} && \
+ CFLAGS="-fPIC -O3" make -j8 && \
+ make install && \
+ rm -rf /zstd-${ZSTD_VERSION} /zstd-${ZSTD_VERSION}.tar.gz
+
+# Snappy
+RUN SNAPPY_VERSION=$(dep-version.py snappy) && \
+ curl -O -L https://github.com/google/snappy/archive/refs/tags/${SNAPPY_VERSION}.tar.gz && \
+ tar xfz ${SNAPPY_VERSION}.tar.gz && \
+ cd snappy-${SNAPPY_VERSION} && \
+ CXXFLAGS="-fPIC -O3" cmake . -DSNAPPY_BUILD_TESTS=OFF -DSNAPPY_BUILD_BENCHMARKS=OFF && \
+ make -j8 && make install && \
+ rm -rf /snappy-${SNAPPY_VERSION} /snappy-${SNAPPY_VERSION}.tar.gz
Review Comment:
```suggestion
rm -rf /snappy-${SNAPPY_VERSION} /${SNAPPY_VERSION}.tar.gz
```
##########
pkg/deb/Dockerfile:
##########
@@ -25,69 +25,93 @@ ARG PLATFORM
# perl is required to install OpenSSL
RUN apt-get update -y && \
- apt-get install -y curl g++ make perl dpkg-dev python3
+ apt-get install -y \
+ curl \
+ g++ \
+ make \
+ perl \
+ dpkg-dev \
+ python3 \
+ python3-pip
+
+RUN pip3 install pyyaml
+
+ADD .build/dependencies.yaml /
+ADD .build/dep-version.py /usr/local/bin
# Download and compile boost
-RUN curl -O -L https://boostorg.jfrog.io/artifactory/main/release/1.79.0/source/boost_1_79_0.tar.gz && \
- tar xfz boost_1_79_0.tar.gz && \
- cd boost_1_79_0 && \
- ./bootstrap.sh && \
- ./b2 address-model=64 cxxflags=-fPIC link=static threading=multi variant=release install && \
- rm -rf /boost_1_79_0.tar.gz /boost_1_79_0
-
-RUN curl -O -L https://github.com/Kitware/CMake/releases/download/v3.24.0/cmake-3.24.0-linux-${PLATFORM}.tar.gz && \
- tar xfz cmake-3.24.0-linux-${PLATFORM}.tar.gz && \
- cp cmake-3.24.0-linux-${PLATFORM}/bin/* /usr/bin/ && \
- cp -r cmake-3.24.0-linux-${PLATFORM}/share/cmake-3.24 /usr/share/ && \
- rm -rf cmake-3.24.0-linux-${PLATFORM} cmake-3.24.0-linux-${PLATFORM}.tar.gz
-
-# Download and copile protoubf
-RUN curl -O -L https://github.com/google/protobuf/releases/download/v3.20.0/protobuf-cpp-3.20.0.tar.gz && \
- tar xfz protobuf-cpp-3.20.0.tar.gz && \
- cd protobuf-3.20.0/ && \
+RUN BOOST_VERSION=$(dep-version.py boost) && \
+ BOOST_VERSION_UNDESRSCORE=$(echo $BOOST_VERSION | sed 's/\./_/g') && \
+ curl -O -L https://boostorg.jfrog.io/artifactory/main/release/${BOOST_VERSION}/source/boost_${BOOST_VERSION_UNDESRSCORE}.tar.gz && \
+ tar xfz boost_${BOOST_VERSION_UNDESRSCORE}.tar.gz && \
+ cd boost_${BOOST_VERSION_UNDESRSCORE} && \
+ ./bootstrap.sh --with-libraries=regex && \
+ ./b2 -d0 address-model=64 cxxflags=-fPIC link=static threading=multi variant=release install && \
+ rm -rf /boost_${BOOST_VERSION_UNDESRSCORE}.tar.gz /boost_${BOOST_VERSION_UNDESRSCORE}
+
+RUN CMAKE_VERSION=$(dep-version.py cmake) && \
+ curl -O -L https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-linux-${PLATFORM}.tar.gz && \
+ tar xfz cmake-${CMAKE_VERSION}-linux-${PLATFORM}.tar.gz && \
+ cp cmake-${CMAKE_VERSION}-linux-${PLATFORM}/bin/* /usr/bin/ && \
+ cp -r cmake-${CMAKE_VERSION}-linux-${PLATFORM}/share/cmake-* /usr/share/ && \
+ rm -rf cmake-${CMAKE_VERSION}-linux-${PLATFORM} cmake-${CMAKE_VERSION}-linux-${PLATFORM}.tar.gz
+
+# Download and compile protobuf
+RUN PROTOBUF_VERSION=$(dep-version.py protobuf) && \
+ curl -O -L https://github.com/google/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-cpp-${PROTOBUF_VERSION}.tar.gz && \
+ tar xfz protobuf-cpp-${PROTOBUF_VERSION}.tar.gz && \
+ cd protobuf-${PROTOBUF_VERSION}/ && \
CXXFLAGS=-fPIC ./configure && \
make -j8 && make install && ldconfig && \
- rm -rf /protobuf-cpp-3.20.0.tar.gz /protobuf-3.20.0
+ rm -rf /protobuf-cpp-${PROTOBUF_VERSION}.tar.gz /protobuf-${PROTOBUF_VERSION}
# ZLib
-RUN curl -O -L https://github.com/madler/zlib/archive/v1.2.12.tar.gz && \
- tar xfz v1.2.12.tar.gz && \
- cd zlib-1.2.12 && \
+RUN ZLIB_VERSION=$(dep-version.py zlib) && \
+ curl -O -L https://github.com/madler/zlib/archive/v${ZLIB_VERSION}.tar.gz && \
+ tar xfz v${ZLIB_VERSION}.tar.gz && \
+ cd zlib-${ZLIB_VERSION} && \
CFLAGS="-fPIC -O3" ./configure && \
- make && make install && \
- rm -rf /v1.2.12.tar.gz /zlib-1.2.12
+ make -j8 && make install && \
+ rm -rf /v${ZLIB_VERSION}.tar.gz /zlib-${ZLIB_VERSION}
# Zstandard
-RUN curl -O -L https://github.com/facebook/zstd/releases/download/v1.3.7/zstd-1.3.7.tar.gz && \
- tar xfz zstd-1.3.7.tar.gz && \
- cd zstd-1.3.7 && \
+RUN ZSTD_VERSION=$(dep-version.py zstd) && \
+ curl -O -L https://github.com/facebook/zstd/releases/download/v${ZSTD_VERSION}/zstd-${ZSTD_VERSION}.tar.gz && \
+ tar xfz zstd-${ZSTD_VERSION}.tar.gz && \
+ cd zstd-${ZSTD_VERSION} && \
CFLAGS="-fPIC -O3" make -j8 && \
make install && \
- rm -rf /zstd-1.3.7 /zstd-1.3.7.tar.gz
+ rm -rf /zstd-${ZSTD_VERSION} /zstd-${ZSTD_VERSION}.tar.gz
# Snappy
-RUN curl -O -L https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz && \
- tar xfz snappy-1.1.3.tar.gz && \
- cd snappy-1.1.3 && \
- CXXFLAGS="-fPIC -O3" ./configure && \
- make && make install && \
- rm -rf /snappy-1.1.3 /snappy-1.1.3.tar.gz
-
-RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_1n.tar.gz && \
- tar xfz OpenSSL_1_1_1n.tar.gz && \
- cd openssl-OpenSSL_1_1_1n/ && \
+RUN SNAPPY_VERSION=$(dep-version.py snappy) && \
+ curl -O -L https://github.com/google/snappy/archive/refs/tags/${SNAPPY_VERSION}.tar.gz && \
+ tar xfz ${SNAPPY_VERSION}.tar.gz && \
+ cd snappy-${SNAPPY_VERSION} && \
+ CXXFLAGS="-fPIC -O3" cmake . -DSNAPPY_BUILD_TESTS=OFF -DSNAPPY_BUILD_BENCHMARKS=OFF && \
+ make -j8 && make install && \
+ rm -rf /snappy-${SNAPPY_VERSION} /snappy-${SNAPPY_VERSION}.tar.gz
Review Comment:
```suggestion
rm -rf /snappy-${SNAPPY_VERSION} /${SNAPPY_VERSION}.tar.gz
```
##########
pkg/rpm/Dockerfile:
##########
@@ -24,71 +24,94 @@ FROM centos:7
ARG PLATFORM
RUN yum update -y && \
- yum install -y gcc-c++ make rpm-build which \
- createrepo libstdc++-static python3
+ yum install -y \
+ gcc-c++ \
+ make \
+ rpm-build \
+ which \
+ createrepo \
+ libstdc++-static \
+ python3
+
+RUN pip3 install pyyaml
+
+ADD .build/dependencies.yaml /
+ADD .build/dep-version.py /usr/local/bin
# Download and compile boost
# GCC 4.8.2 implementation of std::regex is buggy, so we install boost::regex here
-RUN curl -O -L https://boostorg.jfrog.io/artifactory/main/release/1.79.0/source/boost_1_79_0.tar.gz && \
- tar xfz boost_1_79_0.tar.gz && \
- cd boost_1_79_0 && \
+RUN BOOST_VERSION=$(dep-version.py boost) && \
+ echo "BOOST VERSION: '${BOOST_VERSION}'" && \
+ BOOST_VERSION_UNDESRSCORE=$(echo $BOOST_VERSION | sed 's/\./_/g') && \
+ curl -O -L https://boostorg.jfrog.io/artifactory/main/release/${BOOST_VERSION}/source/boost_${BOOST_VERSION_UNDESRSCORE}.tar.gz && \
+ tar xfz boost_${BOOST_VERSION_UNDESRSCORE}.tar.gz && \
+ cd boost_${BOOST_VERSION_UNDESRSCORE} && \
./bootstrap.sh --with-libraries=regex && \
./b2 address-model=64 cxxflags=-fPIC link=static threading=multi variant=release install && \
- rm -rf /boost_1_79_0.tar.gz /boost_1_79_0
+ rm -rf /boost_${BOOST_VERSION_UNDESRSCORE}.tar.gz /boost_${BOOST_VERSION_UNDESRSCORE}
-RUN curl -O -L https://github.com/Kitware/CMake/releases/download/v3.24.0/cmake-3.24.0-linux-${PLATFORM}.tar.gz && \
- tar xfz cmake-3.24.0-linux-${PLATFORM}.tar.gz && \
- cp cmake-3.24.0-linux-${PLATFORM}/bin/* /usr/bin/ && \
- cp -r cmake-3.24.0-linux-${PLATFORM}/share/cmake-3.24 /usr/share/ && \
- rm -rf cmake-3.24.0-linux-${PLATFORM} cmake-3.24.0-linux-${PLATFORM}.tar.gz
+RUN CMAKE_VERSION=$(dep-version.py cmake) && \
+ curl -O -L https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-linux-${PLATFORM}.tar.gz && \
+ tar xfz cmake-${CMAKE_VERSION}-linux-${PLATFORM}.tar.gz && \
+ cp cmake-${CMAKE_VERSION}-linux-${PLATFORM}/bin/* /usr/bin/ && \
+ cp -r cmake-${CMAKE_VERSION}-linux-${PLATFORM}/share/cmake-* /usr/share/ && \
+ rm -rf cmake-${CMAKE_VERSION}-linux-${PLATFORM} cmake-${CMAKE_VERSION}-linux-${PLATFORM}.tar.gz
# Download and copile protoubf
-RUN curl -O -L https://github.com/google/protobuf/releases/download/v3.20.0/protobuf-cpp-3.20.0.tar.gz && \
- tar xfz protobuf-cpp-3.20.0.tar.gz && \
- cd protobuf-3.20.0/ && \
+RUN PROTOBUF_VERSION=$(dep-version.py protobuf) && \
+ curl -O -L https://github.com/google/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-cpp-${PROTOBUF_VERSION}.tar.gz && \
+ tar xfz protobuf-cpp-${PROTOBUF_VERSION}.tar.gz && \
+ cd protobuf-${PROTOBUF_VERSION}/ && \
CXXFLAGS=-fPIC ./configure && \
make -j8 && make install && ldconfig && \
- rm -rf /protobuf-cpp-3.20.0.tar.gz /protobuf-3.20.0
+ rm -rf /protobuf-cpp-${PROTOBUF_VERSION}.tar.gz /protobuf-${PROTOBUF_VERSION}
# ZLib
-RUN curl -O -L https://github.com/madler/zlib/archive/v1.2.12.tar.gz && \
- tar xfz v1.2.12.tar.gz && \
- cd zlib-1.2.12 && \
+RUN ZLIB_VERSION=$(dep-version.py zlib) && \
+ curl -O -L https://github.com/madler/zlib/archive/v${ZLIB_VERSION}.tar.gz && \
+ tar xfz v${ZLIB_VERSION}.tar.gz && \
+ cd zlib-${ZLIB_VERSION} && \
CFLAGS="-fPIC -O3" ./configure && \
- make && make install && \
- rm -rf /v1.2.12.tar.gz /zlib-1.2.12
+ make -j8 && make install && \
+ rm -rf /v${ZLIB_VERSION}.tar.gz /zlib-${ZLIB_VERSION}
# Zstandard
-RUN curl -O -L https://github.com/facebook/zstd/releases/download/v1.3.7/zstd-1.3.7.tar.gz && \
- tar xfz zstd-1.3.7.tar.gz && \
- cd zstd-1.3.7 && \
+RUN ZSTD_VERSION=$(dep-version.py zstd) && \
+ curl -O -L https://github.com/facebook/zstd/releases/download/v${ZSTD_VERSION}/zstd-${ZSTD_VERSION}.tar.gz && \
+ tar xfz zstd-${ZSTD_VERSION}.tar.gz && \
+ cd zstd-${ZSTD_VERSION} && \
CFLAGS="-fPIC -O3" make -j8 && \
make install && \
- rm -rf /zstd-1.3.7 /zstd-1.3.7.tar.gz
+ rm -rf /zstd-${ZSTD_VERSION} /zstd-${ZSTD_VERSION}.tar.gz
# Snappy
-RUN curl -O -L https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz && \
- tar xfz snappy-1.1.3.tar.gz && \
- cd snappy-1.1.3 && \
- CXXFLAGS="-fPIC -O3" ./configure && \
- make && make install && \
- rm -rf /snappy-1.1.3 /snappy-1.1.3.tar.gz
-
-RUN curl -O -L https://github.com/openssl/openssl/archive/OpenSSL_1_1_1n.tar.gz && \
- tar xfz OpenSSL_1_1_1n.tar.gz && \
- cd openssl-OpenSSL_1_1_1n/ && \
+RUN SNAPPY_VERSION=$(dep-version.py snappy) && \
+ curl -O -L https://github.com/google/snappy/archive/refs/tags/${SNAPPY_VERSION}.tar.gz && \
+ tar xfz ${SNAPPY_VERSION}.tar.gz && \
+ cd snappy-${SNAPPY_VERSION} && \
+ CXXFLAGS="-fPIC -O3" cmake . -DSNAPPY_BUILD_TESTS=OFF -DSNAPPY_BUILD_BENCHMARKS=OFF && \
+ make -j8 && make install && \
+ rm -rf /snappy-${SNAPPY_VERSION} /snappy-${SNAPPY_VERSION}.tar.gz
Review Comment:
```suggestion
rm -rf /snappy-${SNAPPY_VERSION} /${SNAPPY_VERSION}.tar.gz
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-cpp] merlimat merged pull request #19: Build images with deps versions
Posted by GitBox <gi...@apache.org>.
merlimat merged PR #19:
URL: https://github.com/apache/pulsar-client-cpp/pull/19
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org