You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2022/01/19 18:07:15 UTC
Build failed in Jenkins: beam_PostCommit_Py_ValCont #8756
See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/8756/display/redirect?page=changes>
Changes:
[noreply] [BEAM-13015] Provide caching statistics in the status client. (#16495)
[noreply] [BEAM-13611] Skip test_xlang_jdbc_write (#16554)
[noreply] Merge pull request #16370 from [BEAM-13556] playground - color and scroll tabs with new content
[noreply] Merge pull request #16531 from [BEAM-13567] [playground] Handle run code validation and preparation errors
[noreply] Merge pull request #16533 from [BEAM-13548] [Playground] Add example description popover
[noreply] Merge pull request #16519 from [BEAM-13639] [Playground] Add notification to Log/Output tabs about cached example
[noreply] Merge pull request #16518 from [BEAM-13619] [Playground] Add loading animation to the catalog
------------------------------------------
[...truncated 37.15 KB...]
Successfully installed apache-beam-2.37.0.dev0 atomicwrites-1.4.0 attrs-21.4.0 azure-core-1.21.1 azure-storage-blob-12.9.0 boto3-1.20.38 botocore-1.23.38 cachetools-4.2.4 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.10 cloudpickle-2.0.0 crcmod-1.7 cryptography-36.0.1 deprecation-2.1.0 dill-0.3.1.1 docker-5.0.3 docopt-0.6.2 execnet-1.9.0 fastavro-1.4.9 fasteners-0.17.2 freezegun-1.1.0 google-api-core-1.31.5 google-apitools-0.5.31 google-auth-1.35.0 google-cloud-bigquery-2.32.0 google-cloud-bigquery-storage-2.11.0 google-cloud-bigtable-1.7.0 google-cloud-core-1.7.2 google-cloud-datastore-1.15.3 google-cloud-dlp-3.5.0 google-cloud-language-1.3.0 google-cloud-pubsub-2.9.0 google-cloud-pubsublite-1.3.0 google-cloud-recommendations-ai-0.2.0 google-cloud-spanner-1.19.1 google-cloud-videointelligence-1.16.1 google-cloud-vision-1.0.0 google-crc32c-1.3.0 google-resumable-media-2.1.0 googleapis-common-protos-1.54.0 greenlet-1.1.2 grpc-google-iam-v1-0.12.3 grpcio-gcp-0.2.2 grpcio-status-1.43.0 hdfs-2.6.0 httplib2-0.19.1 idna-3.3 isodate-0.6.1 jmespath-0.10.0 libcst-0.4.0 mock-2.0.0 more-itertools-8.12.0 msrest-0.6.21 mypy-extensions-0.4.3 numpy-1.21.5 oauth2client-4.1.3 oauthlib-3.1.1 orjson-3.6.5 overrides-6.1.0 pandas-1.3.5 parameterized-0.7.5 pbr-5.8.0 pluggy-0.13.1 proto-plus-1.19.8 psycopg2-binary-2.9.3 pyarrow-6.0.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pydot-1.4.2 pyhamcrest-1.10.1 pymongo-3.12.3 pyparsing-2.4.7 pytest-4.6.11 pytest-forked-1.4.0 pytest-timeout-1.4.2 pytest-xdist-1.34.0 python-dateutil-2.8.2 pytz-2021.3 pyyaml-6.0 requests-2.27.1 requests-mock-1.9.3 requests-oauthlib-1.3.0 rsa-4.8 s3transfer-0.5.0 sqlalchemy-1.4.29 tenacity-5.1.5 testcontainers-3.4.2 typing-extensions-4.0.1 typing-inspect-0.7.1 typing-utils-0.1.0 urllib3-1.26.8 wcwidth-0.2.5 websocket-client-1.2.3 wrapt-1.13.3
> Task :release:go-licenses:py:dockerRun
+ go-licenses save github.com/apache/beam/sdks/python/container --save_path=/output/licenses
+ go-licenses csv github.com/apache/beam/sdks/python/container
+ tee /output/licenses/list.csv
google.golang.org/genproto/googleapis/rpc/status,https://github.com/googleapis/go-genproto/blob/master/LICENSE,Apache-2.0
golang.org/x/sys,https://go.googlesource.com/sys/+/refs/heads/master/LICENSE,BSD-3-Clause
golang.org/x/text,https://go.googlesource.com/text/+/refs/heads/master/LICENSE,BSD-3-Clause
github.com/apache/beam/sdks/python/container,https://github.com/apache/beam/blob/master/LICENSE,Apache-2.0
google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/master/LICENSE,Apache-2.0
github.com/golang/protobuf,https://github.com/golang/protobuf/blob/master/LICENSE,BSD-3-Clause
google.golang.org/protobuf,https://go.googlesource.com/protobuf/+/refs/heads/master/LICENSE,BSD-3-Clause
github.com/apache/beam/sdks/go/pkg/beam,https://github.com/apache/beam/blob/master/sdks/go/README.md,Apache-2.0
golang.org/x/net,https://go.googlesource.com/net/+/refs/heads/master/LICENSE,BSD-3-Clause
github.com/nightlyone/lockfile,https://github.com/nightlyone/lockfile/blob/master/LICENSE,MIT
+ chmod -R a+w /output/licenses
> Task :release:go-licenses:py:createLicenses
> Task :sdks:python:container:py36:copyGolangLicenses
> Task :sdks:python:container:py38:copyGolangLicenses
> Task :sdks:python:container:py37:copyGolangLicenses
> Task :sdks:python:container:py36:dockerPrepare
> Task :sdks:python:container:py38:dockerPrepare
> Task :sdks:python:container:py37:dockerPrepare
> Task :sdks:python:container:py36:docker
> Task :sdks:python:container:py38:docker
> Task :sdks:python:container:py37:docker
> Task :sdks:python:test-suites:dataflow:py38:validatesContainer
# Where to store integration test outputs.
GCS_LOCATION=${GCS_LOCATION:-gs://temp-storage-for-end-to-end-tests}
# Project for the container and integration test
PROJECT=${PROJECT:-apache-beam-testing}
REGION=${REGION:-us-central1}
IMAGE_PREFIX="$(grep 'docker_image_default_repo_prefix' gradle.properties | cut -d'=' -f2)"
SDK_VERSION="$(grep 'sdk_version' gradle.properties | cut -d'=' -f2)"
# Other variables branched by Python version.
if [[ $1 == "python36" ]]; then
IMAGE_NAME="${IMAGE_PREFIX}python3.6_sdk" # Use this to create CONTAINER_IMAGE variable.
CONTAINER_PROJECT="sdks:python:container:py36" # Use this to build container by Gradle.
PY_INTERPRETER="python3.6" # Use this in virtualenv command.
elif [[ $1 == "python37" ]]; then
IMAGE_NAME="${IMAGE_PREFIX}python3.7_sdk" # Use this to create CONTAINER_IMAGE variable.
CONTAINER_PROJECT="sdks:python:container:py37" # Use this to build container by Gradle.
PY_INTERPRETER="python3.7" # Use this in virtualenv command.
elif [[ $1 == "python38" ]]; then
IMAGE_NAME="${IMAGE_PREFIX}python3.8_sdk" # Use this to create CONTAINER_IMAGE variable.
CONTAINER_PROJECT="sdks:python:container:py38" # Use this to build container by Gradle.
PY_INTERPRETER="python3.8" # Use this in virtualenv command.
else
echo "Must set Python version with one of 'python36', 'python37' and 'python38' from commandline."
exit 1
fi
XUNIT_FILE="pytest-$IMAGE_NAME.xml"
# Verify in the root of the repository
test -d sdks/python/container
# Verify docker and gcloud commands exist
command -v docker
command -v gcloud
docker -v
gcloud -v
# Verify docker image has been built.
docker images | grep "apache/$IMAGE_NAME" | grep "$SDK_VERSION"
TAG=$(date +%Y%m%d-%H%M%S%N)
CONTAINER=us.gcr.io/$PROJECT/$USER/$IMAGE_NAME
PREBUILD_SDK_CONTAINER_REGISTRY_PATH=us.gcr.io/$PROJECT/$USER/prebuild_$1_sdk
echo "Using container $CONTAINER"
# Tag the docker container.
docker tag "apache/$IMAGE_NAME:$SDK_VERSION" "$CONTAINER:$TAG"
# Push the container.
gcloud docker -- push $CONTAINER:$TAG
WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03.
As an alternative, use `gcloud auth configure-docker` to configure `docker` to
use `gcloud` as a credential helper, then use `docker` as you would for non-GCR
registries, e.g. `docker pull gcr.io/project-id/my-image`. Add
`--verbosity=error` to silence this warning: `gcloud docker
--verbosity=error -- pull gcr.io/project-id/my-image`.
See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker
810c583e0ed2: Pushed
2bb4613f59ce: Pushed
20220119-173119282664148: digest: sha256:a21a1ea91637c6d3c6346ac703fa0cb6ec6d3caacb0e94cfccd6f5007e775a3d size: 5361
function cleanup_container {
# Delete the container locally and remotely
docker rmi $CONTAINER:$TAG || echo "Failed to remove container image"
for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH)
do docker rmi $image || echo "Failed to remove prebuilt sdk container image"
done
gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container"
for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk --format="get(digest)")
do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image"
done
>>> Successfully built and push container us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk
>>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST
echo "Removed the container"
}
trap cleanup_container EXIT
echo ">>> Successfully built and push container $CONTAINER"
cd sdks/python
SDK_LOCATION=$2
# Run ValidatesRunner tests on Google Cloud Dataflow service
echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST"
pytest -o junit_suite_name=$IMAGE_NAME \
-m="it_validatescontainer" \
--show-capture=no \
--numprocesses=1 \
--timeout=900 \
--junitxml=$XUNIT_FILE \
--ignore-glob '.*py3\d?\.py$' \
--log-cli-level=INFO \
--test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--region=$REGION \
--sdk_container_image=$CONTAINER:$TAG \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1 \
--prebuild_sdk_container_base_image=$CONTAINER:$TAG \
--docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH"
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-4.6.11, py-1.11.0, pluggy-0.13.1
rootdir: <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python,> inifile: pytest.ini
plugins: xdist-1.34.0, timeout-1.4.2, forked-1.4.0, requests-mock-1.9.3
timeout: 900.0s
timeout method: signal
timeout func_only: False
gw0 I
[gw0] Python 3.8.10 (default, Nov 26 2021, 20:14:08) -- [GCC 9.3.0]
gw0 [4]
scheduling tests via LoadScheduling
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_fnapi_it
[gw0] [ 25%] PASSED apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_fnapi_it
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_cloud_build
[gw0] [ 50%] PASSED apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_cloud_build
apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_local_docker
[gw0] [ 75%] PASSED apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_local_docker
apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it
[gw0] [100%] PASSED apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/pytest-beam_python3.8_sdk.xml> -
=================== 4 passed, 3 warnings in 2014.21 seconds ====================
echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST"
cleanup_container
>>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST
Untagged: us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20220119-173119282664148
Untagged: us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:a21a1ea91637c6d3c6346ac703fa0cb6ec6d3caacb0e94cfccd6f5007e775a3d
Untagged: us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:f72c8590-aa1a-4761-a1b2-b14bf7a73320
Untagged: us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:815bf885c33a927736321bd18899c372216c8449e290c2cc1dbac3de0f250417
Deleted: sha256:5f4479c65601073d23aa27ba7c6962f0b039c630f3bbec92736fa6b8c47813b4
Deleted: sha256:51428d4dee87bd56e0c7146357bf7f080e5a19564efabb629c90f8563f2a4fff
Deleted: sha256:aae4c60fa267ef9631658f0786a79ed3d9780283a479c10a7924b377f8eec953
Deleted: sha256:dd98ead4811294ee14007d226252bc369286ec227d66ca7bef02b217c18193ab
Deleted: sha256:7633bbf970b1be97f8510f62d8659ad2af9872eba5efd66b3712a124a3369933
Deleted: sha256:076c4cdd76a14f3dab9813db838ff435e453571627f779544b42a52cc489f926
Digests:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:a21a1ea91637c6d3c6346ac703fa0cb6ec6d3caacb0e94cfccd6f5007e775a3d
Associated tags:
- 20220119-173119282664148
Tags:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20220119-173119282664148
Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20220119-173119282664148].
Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:a21a1ea91637c6d3c6346ac703fa0cb6ec6d3caacb0e94cfccd6f5007e775a3d].
Digests:
- us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:815bf885c33a927736321bd18899c372216c8449e290c2cc1dbac3de0f250417
Associated tags:
- f72c8590-aa1a-4761-a1b2-b14bf7a73320
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:f72c8590-aa1a-4761-a1b2-b14bf7a73320].
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:815bf885c33a927736321bd18899c372216c8449e290c2cc1dbac3de0f250417].
Digests:
- us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:4c337a7ebe60e5af588785b1642eb3b8586a7c835db84d4a6b96a724c78632e7
Associated tags:
- 0a3364b3-e4ba-45e6-8836-c2897b224da7
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:0a3364b3-e4ba-45e6-8836-c2897b224da7].
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:4c337a7ebe60e5af588785b1642eb3b8586a7c835db84d4a6b96a724c78632e7].
Removed the container
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py36:setupVirtualenv'.
> A problem occurred starting process 'command 'python3.6''
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:setupVirtualenv'.
> A problem occurred starting process 'command 'python3.7''
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
BUILD FAILED in 52m 58s
47 actionable tasks: 37 executed, 4 from cache, 6 up-to-date
Publishing build scan...
https://gradle.com/s/p3vojpiihpzii
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_Py_ValCont #8757
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/8757/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org