You are viewing a plain text version of this content. The canonical link for it is here.

Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2022/03/18 00:53:18 UTC

Build failed in Jenkins: beam_PostCommit_Py_ValCont #8973

See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/8973/display/redirect?page=changes>

Changes:

[noreply] Populate environment capabilities in v1beta3 protos. (#17042)

[noreply] [BEAM-14038] Auto-startup for Python expansion service. (#17035)


------------------------------------------
[...truncated 85.14 KB...]
+ go-licenses save github.com/apache/beam/sdks/python/container --save_path=/output/licenses
+ go-licenses csv github.com/apache/beam/sdks/python/container
+ tee /output/licenses/list.csv
golang.org/x/net,https://go.googlesource.com/net/+/refs/heads/master/LICENSE,BSD-3-Clause
golang.org/x/sys,https://go.googlesource.com/sys/+/refs/heads/master/LICENSE,BSD-3-Clause
github.com/golang/protobuf,https://github.com/golang/protobuf/blob/master/LICENSE,BSD-3-Clause
google.golang.org/genproto/googleapis/rpc/status,https://github.com/googleapis/go-genproto/blob/master/LICENSE,Apache-2.0
golang.org/x/text,https://go.googlesource.com/text/+/refs/heads/master/LICENSE,BSD-3-Clause
github.com/nightlyone/lockfile,https://github.com/nightlyone/lockfile/blob/master/LICENSE,MIT
github.com/apache/beam/sdks/python/container,https://github.com/apache/beam/blob/master/LICENSE,Apache-2.0
github.com/apache/beam/sdks/go/pkg/beam,https://github.com/apache/beam/blob/master/sdks/go/README.md,Apache-2.0
google.golang.org/grpc,https://github.com/grpc/grpc-go/blob/master/LICENSE,Apache-2.0
google.golang.org/protobuf,https://go.googlesource.com/protobuf/+/refs/heads/master/LICENSE,BSD-3-Clause
+ chmod -R a+w /output/licenses

> Task :release:go-licenses:py:createLicenses
> Task :sdks:python:container:py37:copyGolangLicenses
> Task :sdks:python:container:py39:copyGolangLicenses
> Task :sdks:python:container:py38:copyGolangLicenses
> Task :sdks:python:container:py36:copyGolangLicenses
> Task :sdks:python:container:py39:dockerPrepare
> Task :sdks:python:container:py38:dockerPrepare
> Task :sdks:python:container:py37:dockerPrepare
> Task :sdks:python:container:py36:dockerPrepare
> Task :sdks:python:container:py39:docker
> Task :sdks:python:container:py38:docker
> Task :sdks:python:container:py37:docker
> Task :sdks:python:container:py36:docker

> Task :sdks:python:container:py39:docker
The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then       pip install 'pip-licenses<3.0.0' pyyaml tenacity &&       python /tmp/license_scripts/pull_licenses_py.py ;     fi' returned a non-zero code: 1

> Task :sdks:python:container:py39:docker FAILED

> Task :sdks:python:container:py38:docker
[91mTraceback (most recent call last):
  File "/tmp/license_scripts/pull_licenses_py.py", line 166, in <module>
    dependencies = run_pip_licenses()
  File "/tmp/license_scripts/pull_licenses_py.py", line 49, in run_pip_licenses
    dependencies = run_bash_command(command)
  File "/tmp/license_scripts/pull_licenses_py.py", line 44, in run_bash_command
    return subprocess.check_output(command.split()).decode('utf-8')
  File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pip-licenses', '--with-license-file', '--with-urls', '--from=mixed', '--ignore', 'apache-beam', '--format=json']' returned non-zero exit status 1.
[0m
> Task :sdks:python:container:py37:docker
[91mTraceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/piplicenses.py", line 40, in <module>
    from pip._internal.utils.misc import get_installed_distributions
ImportError: cannot import name 'get_installed_distributions' from 'pip._internal.utils.misc' (/usr/local/lib/python3.7/site-packages/pip/_internal/utils/misc.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pip-licenses", line 5, in <module>
    from piplicenses import main
  File "/usr/local/lib/python3.7/site-packages/piplicenses.py", line 42, in <module>
    from pip import get_installed_distributions
ImportError: cannot import name 'get_installed_distributions' from 'pip' (/usr/local/lib/python3.7/site-packages/pip/__init__.py)
[0m[91mTraceback (most recent call last):
  File "/tmp/license_scripts/pull_licenses_py.py", line 166, in <module>
    dependencies = run_pip_licenses()
  File "/tmp/license_scripts/pull_licenses_py.py", line 49, in run_pip_licenses
    dependencies = run_bash_command(command)
  File "/tmp/license_scripts/pull_licenses_py.py", line 44, in run_bash_command
    return subprocess.check_output(command.split()).decode('utf-8')
  File "/usr/local/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/usr/local/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pip-licenses', '--with-license-file', '--with-urls', '--from=mixed', '--ignore', 'apache-beam', '--format=json']' returned non-zero exit status 1.
[0m
> Task :sdks:python:container:py36:docker
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/site-packages (from tenacity) (1.15.0)
Building wheels for collected packages: PTable
  Building wheel for PTable (setup.py): started

> Task :sdks:python:container:py38:docker FAILED
The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then       pip install 'pip-licenses<3.0.0' pyyaml tenacity &&       python /tmp/license_scripts/pull_licenses_py.py ;     fi' returned a non-zero code: 1

> Task :sdks:python:container:py37:docker FAILED
The command '/bin/sh -c if [ "$pull_licenses" = "true" ] ; then       pip install 'pip-licenses<3.0.0' pyyaml tenacity &&       python /tmp/license_scripts/pull_licenses_py.py ;     fi' returned a non-zero code: 1

> Task :sdks:python:test-suites:dataflow:py36:validatesContainer

# Where to store integration test outputs.
GCS_LOCATION=${GCS_LOCATION:-gs://temp-storage-for-end-to-end-tests}

# Project for the container and integration test
PROJECT=${PROJECT:-apache-beam-testing}
REGION=${REGION:-us-central1}
IMAGE_PREFIX="$(grep 'docker_image_default_repo_prefix' gradle.properties | cut -d'=' -f2)"
SDK_VERSION="$(grep 'sdk_version' gradle.properties | cut -d'=' -f2)"
PY_VERSION=$1
IMAGE_NAME="${IMAGE_PREFIX}python${PY_VERSION}_sdk"
CONTAINER_PROJECT="sdks:python:container:py${PY_VERSION//.}"  # Note: we substitute away the dot in the version.
PY_INTERPRETER="python${PY_VERSION}"

XUNIT_FILE="pytest-$IMAGE_NAME.xml"

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
command -v gcloud
docker -v
gcloud -v

# Verify docker image has been built.
docker images | grep "apache/$IMAGE_NAME" | grep "$SDK_VERSION"

TAG=$(date +%Y%m%d-%H%M%S%N)
CONTAINER=us.gcr.io/$PROJECT/$USER/$IMAGE_NAME
PREBUILD_SDK_CONTAINER_REGISTRY_PATH=us.gcr.io/$PROJECT/$USER/prebuild_python${PY_VERSION//.}_sdk
echo "Using container $CONTAINER"

# Tag the docker container.
docker tag "apache/$IMAGE_NAME:$SDK_VERSION" "$CONTAINER:$TAG"

# Push the container.
gcloud docker -- push $CONTAINER:$TAG
WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03.

As an alternative, use `gcloud auth configure-docker` to configure `docker` to
use `gcloud` as a credential helper, then use `docker` as you would for non-GCR
registries, e.g. `docker pull gcr.io/project-id/my-image`. Add
`--verbosity=error` to silence this warning: `gcloud docker
--verbosity=error -- pull gcr.io/project-id/my-image`.

See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker


function cleanup_container {
  # Delete the container locally and remotely
  docker rmi $CONTAINER:$TAG || echo "Failed to remove container image"
  for image in $(docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH)
    do docker rmi $image || echo "Failed to remove prebuilt sdk container image"
  done
  gcloud --quiet container images delete $CONTAINER:$TAG || echo "Failed to delete container"
  for digest in $(gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk  --format="get(digest)")
    do gcloud container images delete $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk@$digest --force-delete-tags --quiet || echo "Failed to remove prebuilt sdk container image"
  done

  echo "Removed the container"
}
trap cleanup_container EXIT

echo ">>> Successfully built and push container $CONTAINER"

cd sdks/python
SDK_LOCATION=$2

# Run ValidatesRunner tests on Google Cloud Dataflow service
echo ">>> RUNNING DATAFLOW RUNNER VALIDATESCONTAINER TEST"
pytest -o junit_suite_name=$IMAGE_NAME \
  -m="it_validatescontainer" \
  --show-capture=no \
  --numprocesses=1 \
  --timeout=900 \
  --junitxml=$XUNIT_FILE \
  --ignore-glob '.*py3\d?\.py$' \
  --log-cli-level=INFO \
  --test-pipeline-options=" \
    --runner=TestDataflowRunner \
    --project=$PROJECT \
    --region=$REGION \
    --sdk_container_image=$CONTAINER:$TAG \
    --staging_location=$GCS_LOCATION/staging-validatesrunner-test \
    --temp_location=$GCS_LOCATION/temp-validatesrunner-test \
    --output=$GCS_LOCATION/output \
    --sdk_location=$SDK_LOCATION \
    --num_workers=1 \
    --docker_registry_push_url=$PREBUILD_SDK_CONTAINER_REGISTRY_PATH"

echo ">>> SUCCESS DATAFLOW RUNNER VALIDATESCONTAINER TEST"
cleanup_container
Digests:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:df3bbe462b5f42534d74c4c4c312fccf6f806c654d08dd639b7c7f91dad0516f
  Associated tags:
 - 20220318-001635787377576
Tags:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220318-001635787377576
Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20220318-001635787377576].
Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:df3bbe462b5f42534d74c4c4c312fccf6f806c654d08dd639b7c7f91dad0516f].
Digests:
- us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:efd2b53173e84e01712aba1cb5cf4d863b1a0a524db24e21ece1ac73f56450a9
  Associated tags:
 - 44306aef-85f9-4b21-8c97-a68934025ba2
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:44306aef-85f9-4b21-8c97-a68934025ba2].
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:efd2b53173e84e01712aba1cb5cf4d863b1a0a524db24e21ece1ac73f56450a9].
Digests:
- us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:979e117a03c7a0a6c0f9658d389c1d1febac58ba5ab84795e076a1498de5acd6
  Associated tags:
 - 49786744-442c-43c6-9f56-e3c31f9cc939
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:49786744-442c-43c6-9f56-e3c31f9cc939].
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:979e117a03c7a0a6c0f9658d389c1d1febac58ba5ab84795e076a1498de5acd6].

FAILURE: Build completed with 3 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':sdks:python:container:py39:docker'.
> Process 'command 'docker'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':sdks:python:container:py38:docker'.
> Process 'command 'docker'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':sdks:python:container:py37:docker'.
> Process 'command 'docker'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 52m 45s
59 actionable tasks: 48 executed, 4 from cache, 7 up-to-date

Publishing build scan...
https://gradle.com/s/r4ctidge4e6y6

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org