You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/07/21 21:27:12 UTC
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request, #1374: MINIFICPP-1888 Increase timeout for centos CI build
lordgamez opened a new pull request, #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374
With more extensions added to be turned on by default, the centos build targeting the default extensions takes longer to finish. We should increase the timeout as it sometimes exceeds the current 60 minutes.
https://issues.apache.org/jira/browse/MINIFICPP-1888
------------------------------
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on pull request #1374: MINIFICPP-1888 Increase timeout for centos CI build
Posted by GitBox <gi...@apache.org>.
lordgamez commented on PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#issuecomment-1191975307
> I would rather start disabling a few extensions here and there. We should test the core everywhere, but testing everything everywhere might be more excessive than it's worth it.
>
> I would disable AWS, GCP, Kafka, SQL and Civetweb on centos (and maybe on mac os), since they pull in dependency libs that take some time to compile.
I thought the reason for adding new default extensions was to include them in the centos build by default, but then I'm mistaken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adam-markovics commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
Posted by GitBox <gi...@apache.org>.
adam-markovics commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r942387849
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
Review Comment:
I think so. Otherwise I'm okay with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
Posted by GitBox <gi...@apache.org>.
lordgamez commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r941304237
##########
docker/centos/Dockerfile:
##########
@@ -17,33 +17,47 @@
#
# First stage: the build environment
-FROM centos:7 AS build_deps
+FROM centos:7
LABEL maintainer="Apache NiFi <de...@nifi.apache.org>"
ARG MINIFI_VERSION
+ARG UID=1000
+ARG GID=1000
+
+# MINIFI_OPTIONS will be passed directly to cmake
+# use it to define cmake options (e.g. -DENABLE_AWS=ON -DENABLE_AZURE=ON)
+ARG MINIFI_OPTIONS=""
+ARG CMAKE_BUILD_TYPE=Release
+ARG DOCKER_SKIP_TESTS=ON
# Install the system dependencies needed for a build
ENV MINIFI_BASE_DIR /opt/minifi
ENV MINIFI_HOME $MINIFI_BASE_DIR/nifi-minifi-cpp-$MINIFI_VERSION
+ENV USER minificpp
# gpsd-devel is in EPEL
RUN yum -y install epel-release && yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel python36-devel sudo git which maven make libarchive boost-devel lua-devel libusbx-devel libpng-devel \
- gpsd-devel libpcap-devel
+ gpsd-devel libpcap-devel ccache
RUN mkdir -p $MINIFI_BASE_DIR
COPY . ${MINIFI_BASE_DIR}
-FROM build_deps AS release
+RUN cd $MINIFI_BASE_DIR && \
+ ./bootstrap.sh -t && \
+ ln -s /usr/bin/ccache /usr/lib64/ccache/c++
-# MINIFI_OPTIONS will be passed directly to cmake
-# use it to define cmake options (e.g. -DENABLE_AWS=ON -DENABLE_AZURE=ON)
-ARG MINIFI_OPTIONS=""
-ARG CMAKE_BUILD_TYPE=Release
-# Perform the build
-RUN cd $MINIFI_BASE_DIR \
- && ./bootstrap.sh -t \
- && cd build \
- && scl enable devtoolset-10 -- cmake3 -DSTATIC_BUILD= -DSKIP_TESTS=true ${MINIFI_OPTIONS} -DAWS_ENABLE_UNITY_BUILD=OFF -DEXCLUDE_BOOST=ON -DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE}" .. \
- && scl enable devtoolset-10 -- make -j "$(nproc)" package
+# Setup minificpp user
+RUN groupadd -g ${GID} ${USER} && useradd -g ${GID} ${USER} && \
+ chown -R ${USER}:${USER} ${MINIFI_BASE_DIR} && \
+ if [ -d "${MINIFI_BASE_DIR}/.ccache" ]; then mv ${MINIFI_BASE_DIR}/.ccache /home/${USER}/.ccache; fi
+
+USER ${USER}
+# Perform the build
+RUN cd $MINIFI_BASE_DIR && \
+ cd build && \
+ source /opt/rh/devtoolset-10/enable && \
+ export PATH=/usr/lib64/ccache${PATH:+:${PATH}} && \
+ cmake3 -DSTATIC_BUILD= -DSKIP_TESTS=${DOCKER_SKIP_TESTS} ${MINIFI_OPTIONS} -DAWS_ENABLE_UNITY_BUILD=OFF -DEXCLUDE_BOOST=ON -DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE}" .. && \
+ make -j "$(nproc)" package
Review Comment:
I use `source /opt/rh/devtoolset-10/enable` to have it enabled in this context.
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
Review Comment:
In the plain docker build we have multiple stages and use the `--target build` to create the build image then copy the ccache files from that image if needed, then continue with the second stage for the released image. In the other distro builds we don't have separate build and release images, because we don't use those images for running MiNiFi only get the built binaries from it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
Posted by GitBox <gi...@apache.org>.
lordgamez commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r942491369
##########
.github/workflows/ci.yml:
##########
@@ -201,11 +199,19 @@ jobs:
sudo apt install -y ccache
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
- id: build
- run: mkdir build && cd build && cmake -DSTRICT_GSL_CHECKS=AUDIT -DENABLE_KUBERNETES=ON .. && make centos
+ run: |
+ if [ -d ~/.ccache ]; then mv ~/.ccache .; fi
+ mkdir build && cd build && cmake -DUSE_SHARED_LIBS=ON -DCI_BUILD=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON -DENABLE_AWS=ON -DENABLE_AZURE=ON -DENABLE_COAP=ON \
+ -DENABLE_ENCRYPT_CONFIG=ON -DENABLE_GPS=ON -DENABLE_JNI=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_LINTER=ON -DENABLE_MQTT=ON -DENABLE_NANOFI=ON -DENABLE_OPC=ON -DENABLE_OPENCV=ON \
+ -DENABLE_OPENWSMAN=ON -DENABLE_OPS=ON -DENABLE_PYTHON=ON -DENABLE_SENSORS=ON -DENABLE_SFTP=ON -DENABLE_SQL=ON -DENABLE_SYSTEMD=ON -DENABLE_TENSORFLOW=OFF \
+ -DENABLE_USB_CAMERA=ON -DENABLE_SCRIPTING=ON -DENABLE_LUA_SCRIPTING=ON -DENABLE_KUBERNETES=ON -DENABLE_GCP=ON -DENABLE_PROCFS=ON -DENABLE_PROMETHEUS=ON -DENABLE_ELASTICSEARCH=ON \
+ -DDOCKER_SKIP_TESTS=OFF -DDOCKER_BUILD_ONLY=ON -DDOCKER_CCACHE_DUMP_LOCATION=$HOME/.ccache .. && make centos
+ - id: test
+ run: docker run --rm apacheminificpp:$(docker images | grep apacheminificpp | grep centos | awk '{print $2}') bash -c 'cd /opt/minifi/build && make test ARGS="--timeout 300 -j8 --output-on-failure"'
docker_integration_tests:
name: "Docker integration tests"
runs-on: ubuntu-20.04
- timeout-minutes: 120
+ timeout-minutes: 180
Review Comment:
Updated in 87fd6d00ed72d55b17f0d682da239187ce415250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
Posted by GitBox <gi...@apache.org>.
fgerlits commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r942373803
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
Review Comment:
Could we add stages to all Dockerfiles so we can use them in the same way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm closed pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
Posted by GitBox <gi...@apache.org>.
szaszm closed pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
Posted by GitBox <gi...@apache.org>.
fgerlits commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r941271222
##########
.github/workflows/ci.yml:
##########
@@ -201,11 +199,19 @@ jobs:
sudo apt install -y ccache
echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV
- id: build
- run: mkdir build && cd build && cmake -DSTRICT_GSL_CHECKS=AUDIT -DENABLE_KUBERNETES=ON .. && make centos
+ run: |
+ if [ -d ~/.ccache ]; then mv ~/.ccache .; fi
+ mkdir build && cd build && cmake -DUSE_SHARED_LIBS=ON -DCI_BUILD=ON -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON -DENABLE_AWS=ON -DENABLE_AZURE=ON -DENABLE_COAP=ON \
+ -DENABLE_ENCRYPT_CONFIG=ON -DENABLE_GPS=ON -DENABLE_JNI=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_LINTER=ON -DENABLE_MQTT=ON -DENABLE_NANOFI=ON -DENABLE_OPC=ON -DENABLE_OPENCV=ON \
+ -DENABLE_OPENWSMAN=ON -DENABLE_OPS=ON -DENABLE_PYTHON=ON -DENABLE_SENSORS=ON -DENABLE_SFTP=ON -DENABLE_SQL=ON -DENABLE_SYSTEMD=ON -DENABLE_TENSORFLOW=OFF \
+ -DENABLE_USB_CAMERA=ON -DENABLE_SCRIPTING=ON -DENABLE_LUA_SCRIPTING=ON -DENABLE_KUBERNETES=ON -DENABLE_GCP=ON -DENABLE_PROCFS=ON -DENABLE_PROMETHEUS=ON -DENABLE_ELASTICSEARCH=ON \
+ -DDOCKER_SKIP_TESTS=OFF -DDOCKER_BUILD_ONLY=ON -DDOCKER_CCACHE_DUMP_LOCATION=$HOME/.ccache .. && make centos
+ - id: test
+ run: docker run --rm apacheminificpp:$(docker images | grep apacheminificpp | grep centos | awk '{print $2}') bash -c 'cd /opt/minifi/build && make test ARGS="--timeout 300 -j8 --output-on-failure"'
docker_integration_tests:
name: "Docker integration tests"
runs-on: ubuntu-20.04
- timeout-minutes: 120
+ timeout-minutes: 180
Review Comment:
we could increase the timeout on the MacOS build to 180 minutes, too, as I have seen timeouts on that job in my fork, too
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+ echo DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
Review Comment:
What does `DOCKER_BUILDKIT=0/1` do, and why it change here from 0 to 1?
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
Review Comment:
this is a large, almost-duplicate block -- is there any way of avoiding this?
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
Review Comment:
why can't we (or why don't we want to) do the distro-based builds in the same way as the plain docker build?
##########
docker/centos/Dockerfile:
##########
@@ -17,33 +17,47 @@
#
# First stage: the build environment
-FROM centos:7 AS build_deps
+FROM centos:7
LABEL maintainer="Apache NiFi <de...@nifi.apache.org>"
ARG MINIFI_VERSION
+ARG UID=1000
+ARG GID=1000
+
+# MINIFI_OPTIONS will be passed directly to cmake
+# use it to define cmake options (e.g. -DENABLE_AWS=ON -DENABLE_AZURE=ON)
+ARG MINIFI_OPTIONS=""
+ARG CMAKE_BUILD_TYPE=Release
+ARG DOCKER_SKIP_TESTS=ON
# Install the system dependencies needed for a build
ENV MINIFI_BASE_DIR /opt/minifi
ENV MINIFI_HOME $MINIFI_BASE_DIR/nifi-minifi-cpp-$MINIFI_VERSION
+ENV USER minificpp
# gpsd-devel is in EPEL
RUN yum -y install epel-release && yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel python36-devel sudo git which maven make libarchive boost-devel lua-devel libusbx-devel libpng-devel \
- gpsd-devel libpcap-devel
+ gpsd-devel libpcap-devel ccache
RUN mkdir -p $MINIFI_BASE_DIR
COPY . ${MINIFI_BASE_DIR}
-FROM build_deps AS release
+RUN cd $MINIFI_BASE_DIR && \
+ ./bootstrap.sh -t && \
+ ln -s /usr/bin/ccache /usr/lib64/ccache/c++
-# MINIFI_OPTIONS will be passed directly to cmake
-# use it to define cmake options (e.g. -DENABLE_AWS=ON -DENABLE_AZURE=ON)
-ARG MINIFI_OPTIONS=""
-ARG CMAKE_BUILD_TYPE=Release
-# Perform the build
-RUN cd $MINIFI_BASE_DIR \
- && ./bootstrap.sh -t \
- && cd build \
- && scl enable devtoolset-10 -- cmake3 -DSTATIC_BUILD= -DSKIP_TESTS=true ${MINIFI_OPTIONS} -DAWS_ENABLE_UNITY_BUILD=OFF -DEXCLUDE_BOOST=ON -DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE}" .. \
- && scl enable devtoolset-10 -- make -j "$(nproc)" package
+# Setup minificpp user
+RUN groupadd -g ${GID} ${USER} && useradd -g ${GID} ${USER} && \
+ chown -R ${USER}:${USER} ${MINIFI_BASE_DIR} && \
+ if [ -d "${MINIFI_BASE_DIR}/.ccache" ]; then mv ${MINIFI_BASE_DIR}/.ccache /home/${USER}/.ccache; fi
+
+USER ${USER}
+# Perform the build
+RUN cd $MINIFI_BASE_DIR && \
+ cd build && \
+ source /opt/rh/devtoolset-10/enable && \
+ export PATH=/usr/lib64/ccache${PATH:+:${PATH}} && \
+ cmake3 -DSTATIC_BUILD= -DSKIP_TESTS=${DOCKER_SKIP_TESTS} ${MINIFI_OPTIONS} -DAWS_ENABLE_UNITY_BUILD=OFF -DEXCLUDE_BOOST=ON -DCMAKE_BUILD_TYPE="${CMAKE_BUILD_TYPE}" .. && \
+ make -j "$(nproc)" package
Review Comment:
why is `scl enable devtoolset-10` not needed any longer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
Posted by GitBox <gi...@apache.org>.
fgerlits commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r941280092
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+ echo DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
Review Comment:
What does `DOCKER_BUILDKIT=0/1` do, and why did it change here from 0 to 1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
Posted by GitBox <gi...@apache.org>.
lordgamez commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r942389999
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
Review Comment:
I would rather not, stages are usually used if we need one or more helper images to create the final image, this way the layers of the helper stages only created as temporary layers that can be cleaned up afterwards. In the centos and other distro docker images we do not really have any logically distinguished stages and I would remove these stages in the focal, fedora, bionic Dockerfiles as well in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on pull request #1374: MINIFICPP-1888 Increase timeout for centos CI build
Posted by GitBox <gi...@apache.org>.
szaszm commented on PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#issuecomment-1192012841
I did it in #1341, since I thought that these extensions are stable and well tested, there's not much reason a user wouldn't want to build and have them. Released convenience binaries should contain these. If a user doesn't want the functionality, they can just remove the extension .so/.dlls from their installation.
The CI is a different beast, since it's costing the ASF money per machine minute. We can use it, but we should make an effort to not overuse it, and testing extensions once or twice on _a_ linux is probably good enough coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
Posted by GitBox <gi...@apache.org>.
lordgamez commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r941303138
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+ echo DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
Review Comment:
DOCKER_BUILDKIT enables additional features like the copying of files between multiple stages and only building specific stages, that we use in the alpine based docker image. We don't need that in the other distros. I would use it with `DOCKER_BUILDKIT=1` by default, but I'm not sure if there was a problem with that on any build machines previously, as it was changed to 0 in the past.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1374: MINIFICPP-1888 Move all extension builds to CentOS job
Posted by GitBox <gi...@apache.org>.
lordgamez commented on code in PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#discussion_r942403515
##########
docker/DockerBuild.sh:
##########
@@ -146,18 +149,31 @@ BUILD_ARGS+=("--build-arg" "UID=${UID_ARG}"
"--build-arg" "GID=${GID_ARG}"
"--build-arg" "MINIFI_VERSION=${MINIFI_VERSION}"
"--build-arg" "DUMP_LOCATION=${DUMP_LOCATION}"
- "--build-arg" "DISTRO_NAME=${DISTRO_NAME}")
+ "--build-arg" "DISTRO_NAME=${DISTRO_NAME}"
+ "--build-arg" "DOCKER_SKIP_TESTS=${DOCKER_SKIP_TESTS}")
-if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
- DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
+if [ -n "${DISTRO_NAME}" ]; then
+ echo DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
- container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
- mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
- docker rm -f "${container_id}"
-fi
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ container_id=$(docker run --rm -d apacheminificpp:"${TAG}" sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
+else
+ if [ -n "${DOCKER_CCACHE_DUMP_LOCATION}" ]; then
+ DOCKER_BUILDKIT=1 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} --target build -t minifi_build ..
-DOCKER_BUILDKIT=0 docker build "${BUILD_ARGS[@]}" -f ${DOCKERFILE} -t apacheminificpp:"${TAG}" ..
+ container_id=$(docker run --rm -d minifi_build sh -c "while true; do sleep 1; done")
+ mkdir -p "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker cp "${container_id}:/home/minificpp/.ccache/." "${DOCKER_CCACHE_DUMP_LOCATION}"
+ docker rm -f "${container_id}"
+ fi
Review Comment:
Extracted to a separate function in 6af554983dc9bfb2cdff69fd134f909374a876e9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on pull request #1374: MINIFICPP-1888 Increase timeout for centos CI build
Posted by GitBox <gi...@apache.org>.
szaszm commented on PR #1374:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1374#issuecomment-1191969364
I would rather start disabling a few extensions here and there. We should test the core everywhere, but testing everything everywhere might be more excessive than it's worth it.
I would disable AWS, GCP, Kafka and SQL on centos (and maybe on mac os), since they pull in dependency libs that take some time to compile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org