You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/08/09 12:45:12 UTC
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
fgerlits commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r941271222
##########
.github/workflows/ci.yml:
##########
@@ -201,11 +199,19 @@ jobs:
sudo apt install -y ccache
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
- id: build
- run: mkdir build && cd build && cmake -DSTRICT_GSL_CHECKS=AUDIT -DENABLE_KUBERNETES=ON .. && make centos
+ run: |
+ if [ -d ~/.ccache ]; then mv ~/.ccache .; fi
+ mkdir build && cd build && cmake -DUSE_SHARED_LIBS=ON -DCI_BUILD=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON -DENABLE_AWS=ON -DENABLE_AZURE=ON -DENABLE_COAP=ON \
+ -DENABLE_ENCRYPT_CONFIG=ON -DENABLE_GPS=ON -DENABLE_JNI=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_LINTER=ON -DENABLE_MQTT=ON -DENABLE_NANOFI=ON -DENABLE_OPC=ON -DENABLE_OPENCV=ON \
+ -DENABLE_OPENWSMAN=ON -DENABLE_OPS=ON -DENABLE_PYTHON=ON -DENABLE_SENSORS=ON -DENABLE_SFTP=ON -DENABLE_SQL=ON -DENABLE_SYSTEMD=ON -DENABLE_TENSORFLOW=OFF \
+ -DENABLE_USB_CAMERA=ON -DENABLE_SCRIPTING=ON -DENABLE_LUA_SCRIPTING=ON -DENABLE_KUBERNETES=ON -DENABLE_GCP=ON -DENABLE_PROCFS=ON -DENABLE_PROMETHEUS=ON -DENABLE_ELASTICSEARCH=ON \
+ -DDOCKER_SKIP_TESTS=OFF -DDOCKER_BUILD_ONLY=ON -DDOCKER_CCACHE_DUMP_LOCATION=$HOME/.ccache .. && make centos
+ - id: test
+ run: docker run --rm apacheminificpp:$(docker images | grep apacheminificpp | grep centos | awk '{print $2}') bash -c 'cd /opt/minifi/build && make test ARGS="--timeout 300 -j8 --output-on-failure"'
docker_integration_tests:
name: "Docker integration tests"
runs-on: ubuntu-20.04
- timeout-minutes: 120
+ timeout-minutes: 180
Review Comment:
we could increase the timeout on the MacOS build to 180 minutes, too, as I have seen timeouts on that job in my fork, too
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+ echo DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
Review Comment:
What does `DOCKER_BUILDKIT=0/1` do, and why it change here from 0 to 1?
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
Review Comment:
this is a large, almost-duplicate block -- is there any way of avoiding this?
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
Review Comment:
why can't we (or why don't we want to) do the distro-based builds in the same way as the plain docker build?
##########
docker/centos/Dockerfile:
##########
@@ -17,33 +17,47 @@
#
# First stage: the build environment
-FROM centos:7 AS build_deps
+FROM centos:7
LABEL maintainer="Apache NiFi <de...@nifi.apache.org>"
ARG MINIFI_VERSION
+ARG UID=1000
+ARG GID=1000
+
+# MINIFI_OPTIONS will be passed directly to cmake
+# use it to define cmake options (e.g. -DENABLE_AWS=ON -DENABLE_AZURE=ON)
+ARG MINIFI_OPTIONS=""
+ARG CMAKE_BUILD_TYPE=Release
+ARG DOCKER_SKIP_TESTS=ON
# Install the system dependencies needed for a build
ENV MINIFI_BASE_DIR /opt/minifi
ENV MINIFI_HOME $MINIFI_BASE_DIR/nifi-minifi-cpp-$MINIFI_VERSION
+ENV USER minificpp
# gpsd-devel is in EPEL
RUN yum -y install epel-release && yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel python36-devel sudo git which maven make libarchive boost-devel lua-devel libusbx-devel libpng-devel \
- gpsd-devel libpcap-devel
+ gpsd-devel libpcap-devel ccache
RUN mkdir -p $MINIFI_BASE_DIR
COPY . ${MINIFI_BASE_DIR}
-FROM build_deps AS release
+RUN cd $MINIFI_BASE_DIR && \
+ ./bootstrap.sh -t && \
+ ln -s /usr/bin/ccache /usr/lib64/ccache/c++
-# MINIFI_OPTIONS will be passed directly to cmake
-# use it to define cmake options (e.g. -DENABLE_AWS=ON -DENABLE_AZURE=ON)
-ARG MINIFI_OPTIONS=""
-ARG CMAKE_BUILD_TYPE=Release
-# Perform the build
-RUN cd $MINIFI_BASE_DIR \
- && ./bootstrap.sh -t \
- && cd build \
- && scl enable devtoolset-10 -- cmake3 -DSTATIC_BUILD= -DSKIP_TESTS=true ${MINIFI_OPTIONS} -DAWS_ENABLE_UNITY_BUILD=OFF -DEXCLUDE_BOOST=ON -DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE}" .. \
- && scl enable devtoolset-10 -- make -j "$(nproc)" package
+# Setup minificpp user
+RUN groupadd -g ${GID} ${USER} && useradd -g ${GID} ${USER} && \
+ chown -R ${USER}:${USER} ${MINIFI_BASE_DIR} && \
+ if [ -d "${MINIFI_BASE_DIR}/.ccache" ]; then mv ${MINIFI_BASE_DIR}/.ccache /home/${USER}/.ccache; fi
+
+USER ${USER}
+# Perform the build
+RUN cd $MINIFI_BASE_DIR && \
+ cd build && \
+ source /opt/rh/devtoolset-10/enable && \
+ export PATH=/usr/lib64/ccache${PATH:+:${PATH}} && \
+ cmake3 -DSTATIC_BUILD= -DSKIP_TESTS=${DOCKER_SKIP_TESTS} ${MINIFI_OPTIONS} -DAWS_ENABLE_UNITY_BUILD=OFF -DEXCLUDE_BOOST=ON -DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE}" .. && \
+ make -j "$(nproc)" package
Review Comment:
why is `scl enable devtoolset-10` not needed any longer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org