You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@geode.apache.org by GitBox <gi...@apache.org> on 2021/02/17 05:32:14 UTC
[GitHub] [geode-native] mmartell opened a new pull request #746: GEODE-8949: Create a new CI pipeline
mmartell opened a new pull request #746:
URL: https://github.com/apache/geode-native/pull/746
This brings in the work of pivotal-jbarrett to the develop branch. It has been tested after merging with develop. Several new pieces are being brought in including but not limited to:
- a PR pipeline (all PRs must pass this pipeline before they can be merged)
- a new yaml templating tool (ytt)
- new Windows tooling (Visual Studio 2019)
- new Windows operating system (Windows Server 2019)
- enhanced test discovery and test parallelism
To keep the geode-native history clean, the almost 200 commits have been squashed into this single commit, which is highlighted below:
* Decrease parallelism to fix Windows 2019 test failures
* Fixes versioning together
* Remove ABI compatibility flags
* Increase TIMEBOMB to 600
* Disable DataInput test for now
* Fix clang-format error
* Fixes packer image projects
* Rename keys and tokens per RTM names
* Remove Python, PIP and Coveralls from Ubuntu 16.04
* Integrate rat-check properly into GitHub PR apparatus
* Add license and/or exclusion as appropriate to pass RAT check (#4)
* Fix ABI compatibility
* Use LLVM 11
* Add Ubunutu 20.04
* Upgrade cmake and add make
* Remove unused exports. Remaining exports are for testing.
* Hide symbols from staticly linked libraries
* Increase timeouts
* Fix warnings
* Add github icon
* Split out GitHub components for various GitHub resources
* Tag builds and upload artifacts
* Add functions to reduce yaml redundancy
* Reduce resource complexity
* Use resource to expand geode
* Updates for release and pr pipelines
* Fixes per clang-tidy and hide noise
* Fixes per clang-format and hide noise
* Add clang-tools Dockerfile
* Improve CacheImpl (nothing should be virtual)
* Improve abstract classes (should define virtual destructors)
* Use the concourse worker's zone by default
* Don't be so verbose with pipeline output
* Always fetch latest cmake
* Rerun failed tests (ctest retry with count 4)
* Disable broken IPv6 test
* Fix test failures due to folder collisions
* Cleanup test output directories for failed tests
* Improve execution of gfsh (use less memory)
* Improve gfsh scoping in integration tests
* Add GfshExecute class to legacy integration tests
* Add heap settings to GfshExecute
* Kill all user processes before running tests
* Use tar to speed up file copies
* README for GitHub Private Key
* Fix for .NET 3.5 install on Windows 2016
* Fixes boost on windows
* Fixes for ACE
* Don't checkout sources after upload.
* Update Building.md per new tooling support
* Package icon
* Fixes Windows 2016
* Remove used windows build scripts
* Update to boost 1.75.0 and fix MSVC
* Support for Visual Studio 2019
* Explicitly set MSVC toolset and Windows SDK
* Don't retrigger PRs on new versions
* Improve build-in help for set-pipeline.sh
* Don't rebuild on image changes
* Retry packer builds if they fail
* Update README.md for new CI
* Add error for canceled jobs
* Copy source from worker to builder
* Support for PR resource
* Split release and pr pipelines
* Upload packages to GCS
* Parameterize bucket and key
* Configure build type, version, date, etc
* Support semver
* Templatize docker builds
* Use newer registry-image resource for images not built in this project.
* Fix Dockerfile path and caches.
* Use project specific minimal task image
* Fix legacy integration port allocations
* Fix .NET test execution
* Add support for Windows Server 2019
* Fix install of .NET 3.5 via WinRM
* Added Google Compute support
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578605951
##########
File path: ci/lib/templates.lib.txt
##########
@@ -0,0 +1,253 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+(@ load("@ytt:data", "data") -@)
+(@ load("functions.lib.yml",
+ "gci_label_value",
+ "image_family_name")
+-@)
+
+(@ def google_variables(): -@)
+GOOGLE_PROJECT=(@=data.values.google.project @)
+GOOGLE_ZONE=(@=data.values.google.zone @)
+(@- end @)
+
+
+(@ def instance_variables(): -@)
+INSTANCE_USER=build
+INSTANCE_DIR=$(pwd)/instance
+
+if [ ! -d "${INSTANCE_DIR}" ]; then
+ echo "${INSTANCE_DIR} not found."
+ exit 1
+fi
+(@- end @)
+
+(@ def remote_functions(): -@)
+(@=instance_variables() @)
+
+SSH_OPTIONS=${SSH_OPTIONS:-"-o StrictHostKeyChecking=no -o PasswordAuthentication=no"}
+
+ssh_key_file=${INSTANCE_DIR}/identity
+if [ ! -r "${ssh_key_file}" ]; then
+ echo "${ssh_key_file} not readable."
+ exit 1
+fi
+
+instance_file=${INSTANCE_DIR}/instance.sh
+if [ ! -r "${instance_file}" ]; then
+ echo "${instance_file} not readable."
+ exit 1
+fi
+
+external_ip=$(source ${instance_file} && echo -n ${networkInterfaces_accessConfigs_natIP})
+
+function remote_shell {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} "$@"
+}
+
+function remote_download {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "${INSTANCE_USER}@${external_ip}:${1}" "$2"
+}
+
+function remote_download_directory {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} tar -C $(dirname ${1}) -czf - $(basename ${1}) | tar -C ${2} -zxf -
+}
+
+function remote_upload {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "$1" "${INSTANCE_USER}@${external_ip}:${2}"
+}
+
+(@- end @)
+
+(@ def run_cpp_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/test ctest -C ${CMAKE_CONFIG} -j8 --timeout=500 --output-on-failure
Review comment:
Here's an [example](https://github.com/moleske/geode-native/runs/1923184473) where I use `--repeat until-pass:3` and `--schedule-random` while running your PR in my shadow pipeline. It basically just runs the failed test immediately again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] mmartell commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
mmartell commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578817474
##########
File path: docker/centos-7/Dockerfile
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM centos/devtoolset-4-toolchain-centos7:latest
+LABEL maintainer Apache Geode <de...@geode.apache.org>
+
+USER root
+WORKDIR /
+
+RUN yum update -y && \
+ yum -y clean all
+
+COPY bellsoft.repo /etc/yum.repos.d/
+
+RUN yum update -y && \
+ yum install -y \
+ git \
+ make \
+ zlib-devel \
+ patch \
+ openssl-devel \
+ bellsoft-java11 \
+ doxygen \
+ python3-pip \
+ which && \
+ yum -y clean all
+
+RUN pip3 install --upgrade pip && \
+ pip3 install cpp-coveralls
+
+RUN installer=$(mktemp) \
+ && curl -o ${installer} -L $(curl -s https://api.github.com/repos/Kitware/CMake/releases/latest \
+ | grep 'browser_download_url.*Linux-x86_64\.sh' \
+ | cut -d : -f 2,3 \
+ | tr -d '"') \
+ && bash ${installer} --skip-license --prefix=/usr/local \
+ && rm ${installer}
+
+ARG GEODE_VERSION=1.12.0
Review comment:
Captured the GEODE_VERSION issue as GEODE-8954. Also, put in your idea of passing in the variable to the docker command.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske edited a comment on pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske edited a comment on pull request #746:
URL: https://github.com/apache/geode-native/pull/746#issuecomment-781577308
Can we expose the details of jobs on https://concourse.apachegeode-ci.info/teams/main/pipelines/geode-native-develop?group=all ? I'm pretty sure there are no secrets being printed (or at least there shouldn't be.) This way someone who is not a committer yet can see what the jobs are doing. I'm also trying to compare some runs to my shadow CI to understand the code changes related to llvm 11 and visual studio 2019 and don't feel I should have to request concourse access to see this information
Here are the [checks](https://github.com/moleske/geode-native/pull/36/checks) I'm running for those curious
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578603357
##########
File path: ci/lib/templates.lib.txt
##########
@@ -0,0 +1,253 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+(@ load("@ytt:data", "data") -@)
+(@ load("functions.lib.yml",
+ "gci_label_value",
+ "image_family_name")
+-@)
+
+(@ def google_variables(): -@)
+GOOGLE_PROJECT=(@=data.values.google.project @)
+GOOGLE_ZONE=(@=data.values.google.zone @)
+(@- end @)
+
+
+(@ def instance_variables(): -@)
+INSTANCE_USER=build
+INSTANCE_DIR=$(pwd)/instance
+
+if [ ! -d "${INSTANCE_DIR}" ]; then
+ echo "${INSTANCE_DIR} not found."
+ exit 1
+fi
+(@- end @)
+
+(@ def remote_functions(): -@)
+(@=instance_variables() @)
+
+SSH_OPTIONS=${SSH_OPTIONS:-"-o StrictHostKeyChecking=no -o PasswordAuthentication=no"}
+
+ssh_key_file=${INSTANCE_DIR}/identity
+if [ ! -r "${ssh_key_file}" ]; then
+ echo "${ssh_key_file} not readable."
+ exit 1
+fi
+
+instance_file=${INSTANCE_DIR}/instance.sh
+if [ ! -r "${instance_file}" ]; then
+ echo "${instance_file} not readable."
+ exit 1
+fi
+
+external_ip=$(source ${instance_file} && echo -n ${networkInterfaces_accessConfigs_natIP})
+
+function remote_shell {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} "$@"
+}
+
+function remote_download {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "${INSTANCE_USER}@${external_ip}:${1}" "$2"
+}
+
+function remote_download_directory {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} tar -C $(dirname ${1}) -czf - $(basename ${1}) | tar -C ${2} -zxf -
+}
+
+function remote_upload {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "$1" "${INSTANCE_USER}@${external_ip}:${2}"
+}
+
+(@- end @)
+
+(@ def run_cpp_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/test ctest -C ${CMAKE_CONFIG} -j8 --timeout=500 --output-on-failure
+(@- end @)
+
+(@ def run_cpp_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration/test ctest -C ${CMAKE_CONFIG} -j6 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_cpp_legacy_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration-test ctest -C ${CMAKE_CONFIG} -j4 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_net_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/clicache/test2 ../packages/xunit.runner.console.2.4.0/tools/net452/xunit.console.exe ${CMAKE_CONFIG}/Apache.Geode.Tests2.dll -parallel all
Review comment:
Alright makes sense. Here's an [example](https://github.com/moleske/geode-native/runs/1924579941) of how much the verbose output is on cli unit tests when it fails just as a reference
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578608904
##########
File path: docker/centos-7/Dockerfile
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM centos/devtoolset-4-toolchain-centos7:latest
+LABEL maintainer Apache Geode <de...@geode.apache.org>
+
+USER root
+WORKDIR /
+
+RUN yum update -y && \
+ yum -y clean all
+
+COPY bellsoft.repo /etc/yum.repos.d/
+
+RUN yum update -y && \
+ yum install -y \
+ git \
+ make \
+ zlib-devel \
+ patch \
+ openssl-devel \
+ bellsoft-java11 \
+ doxygen \
+ python3-pip \
+ which && \
+ yum -y clean all
+
+RUN pip3 install --upgrade pip && \
+ pip3 install cpp-coveralls
+
+RUN installer=$(mktemp) \
+ && curl -o ${installer} -L $(curl -s https://api.github.com/repos/Kitware/CMake/releases/latest \
+ | grep 'browser_download_url.*Linux-x86_64\.sh' \
+ | cut -d : -f 2,3 \
+ | tr -d '"') \
+ && bash ${installer} --skip-license --prefix=/usr/local \
+ && rm ${installer}
+
+ARG GEODE_VERSION=1.12.0
Review comment:
For the dockerfiles, I'm pretty sure on the `docker build` command you can pass args. So if scripting all those `docker build` commands, you would be able to have a single variable reused
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578033463
##########
File path: docker/centos-7/Dockerfile
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM centos/devtoolset-4-toolchain-centos7:latest
+LABEL maintainer Apache Geode <de...@geode.apache.org>
+
+USER root
+WORKDIR /
+
+RUN yum update -y && \
+ yum -y clean all
+
+COPY bellsoft.repo /etc/yum.repos.d/
+
+RUN yum update -y && \
+ yum install -y \
+ git \
+ make \
+ zlib-devel \
+ patch \
+ openssl-devel \
+ bellsoft-java11 \
+ doxygen \
+ python3-pip \
+ which && \
+ yum -y clean all
+
+RUN pip3 install --upgrade pip && \
+ pip3 install cpp-coveralls
+
+RUN installer=$(mktemp) \
+ && curl -o ${installer} -L $(curl -s https://api.github.com/repos/Kitware/CMake/releases/latest \
+ | grep 'browser_download_url.*Linux-x86_64\.sh' \
+ | cut -d : -f 2,3 \
+ | tr -d '"') \
+ && bash ${installer} --skip-license --prefix=/usr/local \
+ && rm ${installer}
+
+ARG GEODE_VERSION=1.12.0
Review comment:
Why is this not using the latest geode of 1.13.1? This seems to be the case in all of the dockerfiles added. How do we plan to keep this up to date? It seems painful to remember to update this value in all dockerfiles every time there is a release, but maybe that is more of a wait and see problem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] mmartell commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
mmartell commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578595067
##########
File path: ci/lib/templates.lib.txt
##########
@@ -0,0 +1,253 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+(@ load("@ytt:data", "data") -@)
+(@ load("functions.lib.yml",
+ "gci_label_value",
+ "image_family_name")
+-@)
+
+(@ def google_variables(): -@)
+GOOGLE_PROJECT=(@=data.values.google.project @)
+GOOGLE_ZONE=(@=data.values.google.zone @)
+(@- end @)
+
+
+(@ def instance_variables(): -@)
+INSTANCE_USER=build
+INSTANCE_DIR=$(pwd)/instance
+
+if [ ! -d "${INSTANCE_DIR}" ]; then
+ echo "${INSTANCE_DIR} not found."
+ exit 1
+fi
+(@- end @)
+
+(@ def remote_functions(): -@)
+(@=instance_variables() @)
+
+SSH_OPTIONS=${SSH_OPTIONS:-"-o StrictHostKeyChecking=no -o PasswordAuthentication=no"}
+
+ssh_key_file=${INSTANCE_DIR}/identity
+if [ ! -r "${ssh_key_file}" ]; then
+ echo "${ssh_key_file} not readable."
+ exit 1
+fi
+
+instance_file=${INSTANCE_DIR}/instance.sh
+if [ ! -r "${instance_file}" ]; then
+ echo "${instance_file} not readable."
+ exit 1
+fi
+
+external_ip=$(source ${instance_file} && echo -n ${networkInterfaces_accessConfigs_natIP})
+
+function remote_shell {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} "$@"
+}
+
+function remote_download {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "${INSTANCE_USER}@${external_ip}:${1}" "$2"
+}
+
+function remote_download_directory {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} tar -C $(dirname ${1}) -czf - $(basename ${1}) | tar -C ${2} -zxf -
+}
+
+function remote_upload {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "$1" "${INSTANCE_USER}@${external_ip}:${2}"
+}
+
+(@- end @)
+
+(@ def run_cpp_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/test ctest -C ${CMAKE_CONFIG} -j8 --timeout=500 --output-on-failure
Review comment:
Works for me. Looks like this has been in ctest for a long time. Since we do still have a few intermittent failure, It will be interesting to see how random scheduling affects that, and how it uses random scheduling when rerunning failed tests (we currently use --repeat until-pass:4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] mmartell commented on pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
mmartell commented on pull request #746:
URL: https://github.com/apache/geode-native/pull/746#issuecomment-781865930
The Travis build is reporting what looks like a false positive in ACE/OS_NS_netdb.inl during clang-analyzer:
if (0 == name || '\0' == name[0])
return 0;
Others have reported similar false positives. Merging this for now so we can check the rest of the CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on pull request #746:
URL: https://github.com/apache/geode-native/pull/746#issuecomment-780741052
I see one pipeline https://concourse.apachegeode-ci.info/teams/main/pipelines/geode-native-develop?group=all as a running an example of the PR changes. Does this run both the `develop` branch and PRs or is there a separate PR pipeline deployed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578613553
##########
File path: ci/lib/templates.lib.txt
##########
@@ -0,0 +1,253 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+(@ load("@ytt:data", "data") -@)
+(@ load("functions.lib.yml",
+ "gci_label_value",
+ "image_family_name")
+-@)
+
+(@ def google_variables(): -@)
+GOOGLE_PROJECT=(@=data.values.google.project @)
+GOOGLE_ZONE=(@=data.values.google.zone @)
+(@- end @)
+
+
+(@ def instance_variables(): -@)
+INSTANCE_USER=build
+INSTANCE_DIR=$(pwd)/instance
+
+if [ ! -d "${INSTANCE_DIR}" ]; then
+ echo "${INSTANCE_DIR} not found."
+ exit 1
+fi
+(@- end @)
+
+(@ def remote_functions(): -@)
+(@=instance_variables() @)
+
+SSH_OPTIONS=${SSH_OPTIONS:-"-o StrictHostKeyChecking=no -o PasswordAuthentication=no"}
+
+ssh_key_file=${INSTANCE_DIR}/identity
+if [ ! -r "${ssh_key_file}" ]; then
+ echo "${ssh_key_file} not readable."
+ exit 1
+fi
+
+instance_file=${INSTANCE_DIR}/instance.sh
+if [ ! -r "${instance_file}" ]; then
+ echo "${instance_file} not readable."
+ exit 1
+fi
+
+external_ip=$(source ${instance_file} && echo -n ${networkInterfaces_accessConfigs_natIP})
+
+function remote_shell {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} "$@"
+}
+
+function remote_download {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "${INSTANCE_USER}@${external_ip}:${1}" "$2"
+}
+
+function remote_download_directory {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} tar -C $(dirname ${1}) -czf - $(basename ${1}) | tar -C ${2} -zxf -
+}
+
+function remote_upload {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "$1" "${INSTANCE_USER}@${external_ip}:${2}"
+}
+
+(@- end @)
+
+(@ def run_cpp_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/test ctest -C ${CMAKE_CONFIG} -j8 --timeout=500 --output-on-failure --schedule-random
Review comment:
`-j8` on the cppcache unit tests doesn't make sense since there is only one possible job to run. Though fine keeping it if we feel they'll be more
```
2021-02-18 09:27 ~/workspace/geode-native/build/cppcache/test |wip/ci-layers ↓·4 U:1|
moleske-a01 moeng $ ctest -j8
Test project /Users/pivotal/workspace/geode-native/build/cppcache/test
Start 1: apache-geode_unittests
1/1 Test #1: apache-geode_unittests ........... Passed 11.87 sec
100% tests passed, 0 tests failed out of 1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] mmartell commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
mmartell commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578554418
##########
File path: docker/centos-7/Dockerfile
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM centos/devtoolset-4-toolchain-centos7:latest
+LABEL maintainer Apache Geode <de...@geode.apache.org>
+
+USER root
+WORKDIR /
+
+RUN yum update -y && \
+ yum -y clean all
+
+COPY bellsoft.repo /etc/yum.repos.d/
+
+RUN yum update -y && \
+ yum install -y \
+ git \
+ make \
+ zlib-devel \
+ patch \
+ openssl-devel \
+ bellsoft-java11 \
+ doxygen \
+ python3-pip \
+ which && \
+ yum -y clean all
+
+RUN pip3 install --upgrade pip && \
+ pip3 install cpp-coveralls
+
+RUN installer=$(mktemp) \
+ && curl -o ${installer} -L $(curl -s https://api.github.com/repos/Kitware/CMake/releases/latest \
+ | grep 'browser_download_url.*Linux-x86_64\.sh' \
+ | cut -d : -f 2,3 \
+ | tr -d '"') \
+ && bash ${installer} --skip-license --prefix=/usr/local \
+ && rm ${installer}
+
+ARG GEODE_VERSION=1.12.0
Review comment:
Good catch. Yah, we have several places where we set GEODE_VERSION. Seems like this should be set in a single place. For now, setting all to 1.13.1. Will create a ticket to fix this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] mmartell merged pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
mmartell merged pull request #746:
URL: https://github.com/apache/geode-native/pull/746
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578603357
##########
File path: ci/lib/templates.lib.txt
##########
@@ -0,0 +1,253 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+(@ load("@ytt:data", "data") -@)
+(@ load("functions.lib.yml",
+ "gci_label_value",
+ "image_family_name")
+-@)
+
+(@ def google_variables(): -@)
+GOOGLE_PROJECT=(@=data.values.google.project @)
+GOOGLE_ZONE=(@=data.values.google.zone @)
+(@- end @)
+
+
+(@ def instance_variables(): -@)
+INSTANCE_USER=build
+INSTANCE_DIR=$(pwd)/instance
+
+if [ ! -d "${INSTANCE_DIR}" ]; then
+ echo "${INSTANCE_DIR} not found."
+ exit 1
+fi
+(@- end @)
+
+(@ def remote_functions(): -@)
+(@=instance_variables() @)
+
+SSH_OPTIONS=${SSH_OPTIONS:-"-o StrictHostKeyChecking=no -o PasswordAuthentication=no"}
+
+ssh_key_file=${INSTANCE_DIR}/identity
+if [ ! -r "${ssh_key_file}" ]; then
+ echo "${ssh_key_file} not readable."
+ exit 1
+fi
+
+instance_file=${INSTANCE_DIR}/instance.sh
+if [ ! -r "${instance_file}" ]; then
+ echo "${instance_file} not readable."
+ exit 1
+fi
+
+external_ip=$(source ${instance_file} && echo -n ${networkInterfaces_accessConfigs_natIP})
+
+function remote_shell {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} "$@"
+}
+
+function remote_download {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "${INSTANCE_USER}@${external_ip}:${1}" "$2"
+}
+
+function remote_download_directory {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} tar -C $(dirname ${1}) -czf - $(basename ${1}) | tar -C ${2} -zxf -
+}
+
+function remote_upload {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "$1" "${INSTANCE_USER}@${external_ip}:${2}"
+}
+
+(@- end @)
+
+(@ def run_cpp_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/test ctest -C ${CMAKE_CONFIG} -j8 --timeout=500 --output-on-failure
+(@- end @)
+
+(@ def run_cpp_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration/test ctest -C ${CMAKE_CONFIG} -j6 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_cpp_legacy_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration-test ctest -C ${CMAKE_CONFIG} -j4 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_net_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/clicache/test2 ../packages/xunit.runner.console.2.4.0/tools/net452/xunit.console.exe ${CMAKE_CONFIG}/Apache.Geode.Tests2.dll -parallel all
Review comment:
Alright makes sense. Here's an [example](https://github.com/moleske/geode-native/runs/1924579941) of how much the verbose output is on cli unit tests when it fails just as a reference (using this PR and my shadow pipeline)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on pull request #746:
URL: https://github.com/apache/geode-native/pull/746#issuecomment-781577308
Can we expose the details of jobs on https://concourse.apachegeode-ci.info/teams/main/pipelines/geode-native-develop?group=all ? I'm pretty sure there are no secrets being printed (or at least there shouldn't be.) This way someone who is not a committer yet can see what the jobs are doing. I'm also trying to compare some runs to my shadow CI to understand the code changes related to llvm 11 and visual studio 2019 and don't feel I should have to request concourse access to see this information
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] echobravopapa commented on pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
echobravopapa commented on pull request #746:
URL: https://github.com/apache/geode-native/pull/746#issuecomment-781611653
Hi @moleske really appreciate the feedback and feature requests, how would you feel about filing improvement tickets against the CI? the Geode Native community will certainly look to make improvements, but it is valuable that to the CI out there for this to happen...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] mmartell commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
mmartell commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578599761
##########
File path: ci/lib/templates.lib.txt
##########
@@ -0,0 +1,253 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+(@ load("@ytt:data", "data") -@)
+(@ load("functions.lib.yml",
+ "gci_label_value",
+ "image_family_name")
+-@)
+
+(@ def google_variables(): -@)
+GOOGLE_PROJECT=(@=data.values.google.project @)
+GOOGLE_ZONE=(@=data.values.google.zone @)
+(@- end @)
+
+
+(@ def instance_variables(): -@)
+INSTANCE_USER=build
+INSTANCE_DIR=$(pwd)/instance
+
+if [ ! -d "${INSTANCE_DIR}" ]; then
+ echo "${INSTANCE_DIR} not found."
+ exit 1
+fi
+(@- end @)
+
+(@ def remote_functions(): -@)
+(@=instance_variables() @)
+
+SSH_OPTIONS=${SSH_OPTIONS:-"-o StrictHostKeyChecking=no -o PasswordAuthentication=no"}
+
+ssh_key_file=${INSTANCE_DIR}/identity
+if [ ! -r "${ssh_key_file}" ]; then
+ echo "${ssh_key_file} not readable."
+ exit 1
+fi
+
+instance_file=${INSTANCE_DIR}/instance.sh
+if [ ! -r "${instance_file}" ]; then
+ echo "${instance_file} not readable."
+ exit 1
+fi
+
+external_ip=$(source ${instance_file} && echo -n ${networkInterfaces_accessConfigs_natIP})
+
+function remote_shell {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} "$@"
+}
+
+function remote_download {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "${INSTANCE_USER}@${external_ip}:${1}" "$2"
+}
+
+function remote_download_directory {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} tar -C $(dirname ${1}) -czf - $(basename ${1}) | tar -C ${2} -zxf -
+}
+
+function remote_upload {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "$1" "${INSTANCE_USER}@${external_ip}:${2}"
+}
+
+(@- end @)
+
+(@ def run_cpp_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/test ctest -C ${CMAKE_CONFIG} -j8 --timeout=500 --output-on-failure
+(@- end @)
+
+(@ def run_cpp_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration/test ctest -C ${CMAKE_CONFIG} -j6 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_cpp_legacy_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration-test ctest -C ${CMAKE_CONFIG} -j4 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_net_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/clicache/test2 ../packages/xunit.runner.console.2.4.0/tools/net452/xunit.console.exe ${CMAKE_CONFIG}/Apache.Geode.Tests2.dll -parallel all
Review comment:
I have mixed feelings on that. If Concourse supported paginated output then -verbose is fine. But since it doesn't, having so much output sometimes takes too long before you find what you need.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] mmartell commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
mmartell commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578607252
##########
File path: ci/lib/templates.lib.txt
##########
@@ -0,0 +1,253 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+(@ load("@ytt:data", "data") -@)
+(@ load("functions.lib.yml",
+ "gci_label_value",
+ "image_family_name")
+-@)
+
+(@ def google_variables(): -@)
+GOOGLE_PROJECT=(@=data.values.google.project @)
+GOOGLE_ZONE=(@=data.values.google.zone @)
+(@- end @)
+
+
+(@ def instance_variables(): -@)
+INSTANCE_USER=build
+INSTANCE_DIR=$(pwd)/instance
+
+if [ ! -d "${INSTANCE_DIR}" ]; then
+ echo "${INSTANCE_DIR} not found."
+ exit 1
+fi
+(@- end @)
+
+(@ def remote_functions(): -@)
+(@=instance_variables() @)
+
+SSH_OPTIONS=${SSH_OPTIONS:-"-o StrictHostKeyChecking=no -o PasswordAuthentication=no"}
+
+ssh_key_file=${INSTANCE_DIR}/identity
+if [ ! -r "${ssh_key_file}" ]; then
+ echo "${ssh_key_file} not readable."
+ exit 1
+fi
+
+instance_file=${INSTANCE_DIR}/instance.sh
+if [ ! -r "${instance_file}" ]; then
+ echo "${instance_file} not readable."
+ exit 1
+fi
+
+external_ip=$(source ${instance_file} && echo -n ${networkInterfaces_accessConfigs_natIP})
+
+function remote_shell {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} "$@"
+}
+
+function remote_download {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "${INSTANCE_USER}@${external_ip}:${1}" "$2"
+}
+
+function remote_download_directory {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} tar -C $(dirname ${1}) -czf - $(basename ${1}) | tar -C ${2} -zxf -
+}
+
+function remote_upload {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "$1" "${INSTANCE_USER}@${external_ip}:${2}"
+}
+
+(@- end @)
+
+(@ def run_cpp_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/test ctest -C ${CMAKE_CONFIG} -j8 --timeout=500 --output-on-failure
+(@- end @)
+
+(@ def run_cpp_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration/test ctest -C ${CMAKE_CONFIG} -j6 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_cpp_legacy_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration-test ctest -C ${CMAKE_CONFIG} -j4 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_net_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/clicache/test2 ../packages/xunit.runner.console.2.4.0/tools/net452/xunit.console.exe ${CMAKE_CONFIG}/Apache.Geode.Tests2.dll -parallel all
+(@- end @)
+
+(@ def run_net_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/clicache/integration-test2 ../packages/xunit.runner.console.2.4.0/tools/net452/xunit.console.exe ${CMAKE_CONFIG}/Apache.Geode.IntegrationTests2.dll -verbose -maxthreads 6
Review comment:
Very good question. In general, anything that speeds up the CI turnaround is a good thing. I think this is worth investigating, but I don't think it's appropriate for this PR. Jake has made some changes in the area of parallelization that need to be vetted first. In particular, he has a new test discovery mechanism that he's working on. Also, I think we need to fix all test flakiness (and not require --rerun) before doing test optimization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578608904
##########
File path: docker/centos-7/Dockerfile
##########
@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM centos/devtoolset-4-toolchain-centos7:latest
+LABEL maintainer Apache Geode <de...@geode.apache.org>
+
+USER root
+WORKDIR /
+
+RUN yum update -y && \
+ yum -y clean all
+
+COPY bellsoft.repo /etc/yum.repos.d/
+
+RUN yum update -y && \
+ yum install -y \
+ git \
+ make \
+ zlib-devel \
+ patch \
+ openssl-devel \
+ bellsoft-java11 \
+ doxygen \
+ python3-pip \
+ which && \
+ yum -y clean all
+
+RUN pip3 install --upgrade pip && \
+ pip3 install cpp-coveralls
+
+RUN installer=$(mktemp) \
+ && curl -o ${installer} -L $(curl -s https://api.github.com/repos/Kitware/CMake/releases/latest \
+ | grep 'browser_download_url.*Linux-x86_64\.sh' \
+ | cut -d : -f 2,3 \
+ | tr -d '"') \
+ && bash ${installer} --skip-license --prefix=/usr/local \
+ && rm ${installer}
+
+ARG GEODE_VERSION=1.12.0
Review comment:
For the dockerfiles, I'm pretty sure on the `docker build` command you can pass args. So if scripting all those `docker build` commands, you would be able to have a single variable reused. So that can be added to the new jira story to do this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on a change in pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on a change in pull request #746:
URL: https://github.com/apache/geode-native/pull/746#discussion_r578043646
##########
File path: ci/lib/templates.lib.txt
##########
@@ -0,0 +1,253 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+(@ load("@ytt:data", "data") -@)
+(@ load("functions.lib.yml",
+ "gci_label_value",
+ "image_family_name")
+-@)
+
+(@ def google_variables(): -@)
+GOOGLE_PROJECT=(@=data.values.google.project @)
+GOOGLE_ZONE=(@=data.values.google.zone @)
+(@- end @)
+
+
+(@ def instance_variables(): -@)
+INSTANCE_USER=build
+INSTANCE_DIR=$(pwd)/instance
+
+if [ ! -d "${INSTANCE_DIR}" ]; then
+ echo "${INSTANCE_DIR} not found."
+ exit 1
+fi
+(@- end @)
+
+(@ def remote_functions(): -@)
+(@=instance_variables() @)
+
+SSH_OPTIONS=${SSH_OPTIONS:-"-o StrictHostKeyChecking=no -o PasswordAuthentication=no"}
+
+ssh_key_file=${INSTANCE_DIR}/identity
+if [ ! -r "${ssh_key_file}" ]; then
+ echo "${ssh_key_file} not readable."
+ exit 1
+fi
+
+instance_file=${INSTANCE_DIR}/instance.sh
+if [ ! -r "${instance_file}" ]; then
+ echo "${instance_file} not readable."
+ exit 1
+fi
+
+external_ip=$(source ${instance_file} && echo -n ${networkInterfaces_accessConfigs_natIP})
+
+function remote_shell {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} "$@"
+}
+
+function remote_download {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "${INSTANCE_USER}@${external_ip}:${1}" "$2"
+}
+
+function remote_download_directory {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} tar -C $(dirname ${1}) -czf - $(basename ${1}) | tar -C ${2} -zxf -
+}
+
+function remote_upload {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "$1" "${INSTANCE_USER}@${external_ip}:${2}"
+}
+
+(@- end @)
+
+(@ def run_cpp_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/test ctest -C ${CMAKE_CONFIG} -j8 --timeout=500 --output-on-failure
Review comment:
since you're updating to a new cmake, I'd recommend adding `--schedule-random` to all ctest commands (see https://cmake.org/cmake/help/v3.18/manual/ctest.1.html#options)
##########
File path: ci/lib/templates.lib.txt
##########
@@ -0,0 +1,253 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+(@ load("@ytt:data", "data") -@)
+(@ load("functions.lib.yml",
+ "gci_label_value",
+ "image_family_name")
+-@)
+
+(@ def google_variables(): -@)
+GOOGLE_PROJECT=(@=data.values.google.project @)
+GOOGLE_ZONE=(@=data.values.google.zone @)
+(@- end @)
+
+
+(@ def instance_variables(): -@)
+INSTANCE_USER=build
+INSTANCE_DIR=$(pwd)/instance
+
+if [ ! -d "${INSTANCE_DIR}" ]; then
+ echo "${INSTANCE_DIR} not found."
+ exit 1
+fi
+(@- end @)
+
+(@ def remote_functions(): -@)
+(@=instance_variables() @)
+
+SSH_OPTIONS=${SSH_OPTIONS:-"-o StrictHostKeyChecking=no -o PasswordAuthentication=no"}
+
+ssh_key_file=${INSTANCE_DIR}/identity
+if [ ! -r "${ssh_key_file}" ]; then
+ echo "${ssh_key_file} not readable."
+ exit 1
+fi
+
+instance_file=${INSTANCE_DIR}/instance.sh
+if [ ! -r "${instance_file}" ]; then
+ echo "${instance_file} not readable."
+ exit 1
+fi
+
+external_ip=$(source ${instance_file} && echo -n ${networkInterfaces_accessConfigs_natIP})
+
+function remote_shell {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} "$@"
+}
+
+function remote_download {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "${INSTANCE_USER}@${external_ip}:${1}" "$2"
+}
+
+function remote_download_directory {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} tar -C $(dirname ${1}) -czf - $(basename ${1}) | tar -C ${2} -zxf -
+}
+
+function remote_upload {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "$1" "${INSTANCE_USER}@${external_ip}:${2}"
+}
+
+(@- end @)
+
+(@ def run_cpp_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/test ctest -C ${CMAKE_CONFIG} -j8 --timeout=500 --output-on-failure
+(@- end @)
+
+(@ def run_cpp_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration/test ctest -C ${CMAKE_CONFIG} -j6 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_cpp_legacy_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration-test ctest -C ${CMAKE_CONFIG} -j4 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_net_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/clicache/test2 ../packages/xunit.runner.console.2.4.0/tools/net452/xunit.console.exe ${CMAKE_CONFIG}/Apache.Geode.Tests2.dll -parallel all
+(@- end @)
+
+(@ def run_net_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/clicache/integration-test2 ../packages/xunit.runner.console.2.4.0/tools/net452/xunit.console.exe ${CMAKE_CONFIG}/Apache.Geode.IntegrationTests2.dll -verbose -maxthreads 6
Review comment:
Why no `-parallel all`? Or does maxthreads cover that?
##########
File path: ci/lib/templates.lib.txt
##########
@@ -0,0 +1,253 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+(@ load("@ytt:data", "data") -@)
+(@ load("functions.lib.yml",
+ "gci_label_value",
+ "image_family_name")
+-@)
+
+(@ def google_variables(): -@)
+GOOGLE_PROJECT=(@=data.values.google.project @)
+GOOGLE_ZONE=(@=data.values.google.zone @)
+(@- end @)
+
+
+(@ def instance_variables(): -@)
+INSTANCE_USER=build
+INSTANCE_DIR=$(pwd)/instance
+
+if [ ! -d "${INSTANCE_DIR}" ]; then
+ echo "${INSTANCE_DIR} not found."
+ exit 1
+fi
+(@- end @)
+
+(@ def remote_functions(): -@)
+(@=instance_variables() @)
+
+SSH_OPTIONS=${SSH_OPTIONS:-"-o StrictHostKeyChecking=no -o PasswordAuthentication=no"}
+
+ssh_key_file=${INSTANCE_DIR}/identity
+if [ ! -r "${ssh_key_file}" ]; then
+ echo "${ssh_key_file} not readable."
+ exit 1
+fi
+
+instance_file=${INSTANCE_DIR}/instance.sh
+if [ ! -r "${instance_file}" ]; then
+ echo "${instance_file} not readable."
+ exit 1
+fi
+
+external_ip=$(source ${instance_file} && echo -n ${networkInterfaces_accessConfigs_natIP})
+
+function remote_shell {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} "$@"
+}
+
+function remote_download {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "${INSTANCE_USER}@${external_ip}:${1}" "$2"
+}
+
+function remote_download_directory {
+ ssh ${SSH_OPTIONS} -i ${ssh_key_file} ${INSTANCE_USER}@${external_ip} tar -C $(dirname ${1}) -czf - $(basename ${1}) | tar -C ${2} -zxf -
+}
+
+function remote_upload {
+ scp ${SSH_OPTIONS} -i ${ssh_key_file} -q -r "$1" "${INSTANCE_USER}@${external_ip}:${2}"
+}
+
+(@- end @)
+
+(@ def run_cpp_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/test ctest -C ${CMAKE_CONFIG} -j8 --timeout=500 --output-on-failure
+(@- end @)
+
+(@ def run_cpp_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration/test ctest -C ${CMAKE_CONFIG} -j6 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_cpp_legacy_integration_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/cppcache/integration-test ctest -C ${CMAKE_CONFIG} -j4 --timeout=500 --output-on-failure --repeat until-pass:4
+(@- end @)
+
+(@ def run_net_unit_tests(): -@)
+set -ueo pipefail
+(@= remote_functions() @)
+remote_shell cmake -E chdir build/clicache/test2 ../packages/xunit.runner.console.2.4.0/tools/net452/xunit.console.exe ${CMAKE_CONFIG}/Apache.Geode.Tests2.dll -parallel all
Review comment:
I think `-verbose` is still useful even for unit tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode-native] moleske commented on pull request #746: GEODE-8949: Create a new CI pipeline
Posted by GitBox <gi...@apache.org>.
moleske commented on pull request #746:
URL: https://github.com/apache/geode-native/pull/746#issuecomment-781614962
> Hi @moleske really appreciate the feedback and feature requests, how would you feel about filing improvement tickets against the CI? the Geode Native community will certainly look to make improvements, but it is valuable that to the CI out there for this to happen...
I'll unblock, but this has the _Athena burst from Zeus's head_ style (one giant thing all at once) rather than smaller pieces to make it easy to review. Since it is coming in one giant piece, I assumed we were trying to cover all things. Cause I'm noticing now that IPv6 tests pass on linux, mac, and windows but can't find where that is tested in the pipeline (yes it is an optional feature, but we have tests for it and should run it)
My question then, is what kind of review are folks actually looking for then, since there is no direction as to what is important for feedback
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org