You are viewing a plain text version of this content. The canonical link for it is here.

Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2021/10/24 18:36:52 UTC

Build failed in Jenkins: beam_PostCommit_Py_ValCont #8407

See <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/8407/display/redirect>

Changes:


------------------------------------------
[...truncated 111.78 KB...]
[gw0] [ 75%] PASSED apache_beam/examples/wordcount_it_test.py::WordCountIT::test_wordcount_it_with_prebuilt_sdk_container_local_docker 
apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it 
> Task :sdks:python:test-suites:dataflow:py37:validatesContainer

[gw0] [100%] PASSED apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py::ExerciseMetricsPipelineTest::test_metrics_fnapi_it 

=================================== FAILURES ===================================
____ WordCountIT.test_wordcount_it_with_prebuilt_sdk_container_cloud_build _____
[gw0] linux -- Python 3.7.10 <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/build/gradleenv/-1734967052/bin/python3.7>

self = <apache_beam.examples.wordcount_it_test.WordCountIT testMethod=test_wordcount_it_with_prebuilt_sdk_container_cloud_build>

    @pytest.mark.it_validatescontainer
    def test_wordcount_it_with_prebuilt_sdk_container_cloud_build(self):
      self._run_wordcount_it(
          wordcount.run,
          experiment='beam_fn_api',
>         prebuild_sdk_container_engine='cloud_build')

apache_beam/examples/wordcount_it_test.py:67: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apache_beam/examples/wordcount_it_test.py:109: in _run_wordcount_it
    save_main_session=False,
apache_beam/examples/wordcount.py:89: in run
    output | 'Write' >> WriteToText(known_args.output)
apache_beam/pipeline.py:596: in __exit__
    self.result = self.run()
apache_beam/pipeline.py:549: in run
    self._options).run(False)
apache_beam/pipeline.py:573: in run
    return self.runner.run_pipeline(self, self._options)
apache_beam/runners/dataflow/test_dataflow_runner.py:53: in run_pipeline
    self.result = super().run_pipeline(pipeline, options)
apache_beam/runners/dataflow/dataflow_runner.py:459: in run_pipeline
    environments.DockerEnvironment.from_options(options))
apache_beam/transforms/environments.py:341: in from_options
    options)
apache_beam/runners/portability/sdk_container_builder.py:139: in build_container_image
    return builder._build()
apache_beam/runners/portability/sdk_container_builder.py:90: in _build
    self._invoke_docker_build_and_push(container_image_name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <apache_beam.runners.portability.sdk_container_builder._SdkContainerImageCloudBuilder object at 0x7f3f22820f90>
container_image_name = 'us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:2accca16-115d-4a66-80a4-40a64108d127'

    def _invoke_docker_build_and_push(self, container_image_name):
      project_id = self._google_cloud_options.project
      temp_location = self._google_cloud_options.temp_location
      # google cloud build service expects all the build source file to be
      # compressed into a tarball.
      tarball_path = os.path.join(self._temp_src_dir, '%s.tgz' % SOURCE_FOLDER)
      self._make_tarfile(tarball_path, self._temp_src_dir)
      _LOGGER.info(
          "Compressed source files for building sdk container at %s" %
          tarball_path)
    
      container_image_tag = container_image_name.split(':')[-1]
      gcs_location = os.path.join(
          temp_location, '%s-%s.tgz' % (SOURCE_FOLDER, container_image_tag))
      self._upload_to_gcs(tarball_path, gcs_location)
    
      build = cloudbuild.Build()
      build.steps = []
      step = cloudbuild.BuildStep()
      step.name = 'gcr.io/kaniko-project/executor:latest'
      step.args = ['--destination=' + container_image_name, '--cache=true']
      step.dir = SOURCE_FOLDER
    
      build.steps.append(step)
    
      source = cloudbuild.Source()
      source.storageSource = cloudbuild.StorageSource()
      gcs_bucket, gcs_object = self._get_gcs_bucket_and_name(gcs_location)
      source.storageSource.bucket = os.path.join(gcs_bucket)
      source.storageSource.object = gcs_object
      build.source = source
      # TODO(zyichi): make timeout configurable
      build.timeout = '7200s'
    
      now = time.time()
      # operation = client.create_build(project_id=project_id, build=build)
      request = cloudbuild.CloudbuildProjectsBuildsCreateRequest(
          projectId=project_id, build=build)
      build = self._cloudbuild_client.projects_builds.Create(request)
      build_id, log_url = self._get_cloud_build_id_and_log_url(build.metadata)
      _LOGGER.info(
          'Building sdk container with Google Cloud Build, this may '
          'take a few minutes, you may check build log at %s' % log_url)
    
      # block until build finish, if build fails exception will be raised and
      # stops the job submission.
      response = self._cloudbuild_client.projects_builds.Get(
          cloudbuild.CloudbuildProjectsBuildsGetRequest(
              id=build_id, projectId=project_id))
      while response.status in [cloudbuild.Build.StatusValueValuesEnum.QUEUED,
                                cloudbuild.Build.StatusValueValuesEnum.WORKING]:
        time.sleep(10)
        response = self._cloudbuild_client.projects_builds.Get(
            cloudbuild.CloudbuildProjectsBuildsGetRequest(
                id=build_id, projectId=project_id))
    
      if response.status != cloudbuild.Build.StatusValueValuesEnum.SUCCESS:
        raise RuntimeError(
>           'Failed to build python sdk container image on google cloud build, '
            'please check build log for error.')
E       RuntimeError: Failed to build python sdk container image on google cloud build, please check build log for error.

apache_beam/runners/portability/sdk_container_builder.py:287: RuntimeError
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
  <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
    self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf'))  # pylint: disable=anomalous-backslash-in-string

apache_beam/io/filesystems_test.py:62
  <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
    self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'),  # pylint: disable=anomalous-backslash-in-string

-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/pytest-beam_python3.7_sdk.xml> -
============== 1 failed, 3 passed, 2 warnings in 1269.08 seconds ===============
cleanup_container
Untagged: us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20211024-181158382384405
Untagged: us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:633bcbbedad2b914103eba237fd407963850dd42fa52679f6ff8bb48f638b7de
docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH
Untagged: us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:7c5442a1-f2be-4370-85bb-29de1100d4d9
Untagged: us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:e12279e9acbdef9b8813f5df9680bf57f7c39f40009b170066eaec64cc572d0c
Deleted: sha256:e52f7b41323a9beb3151f320e995fa420c0dbe4a04f35c8691c3d2890feda698
Deleted: sha256:14af72680d7c1da21caa56782eda45b72e7b8789ffd56172a2394c316e8efdc1
Deleted: sha256:66dfcbf3112d7fb7959186f4e2a6d560baea4074492ef8ac439e6f695f28858d
Deleted: sha256:dd9f757e882a8c68b971d59de611dafc5970fd7255f890115e363903547ed43f
Deleted: sha256:cf68ff844d0f66288e015e550e5164e4aedc8dd6dfb24e71a59e03a111a50d65
Deleted: sha256:87055f10614a78ebd6f333f72cb84dc175c2ad92fb5b19006991beae676b6bc2
Digests:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:633bcbbedad2b914103eba237fd407963850dd42fa52679f6ff8bb48f638b7de
  Associated tags:
 - 20211024-181158382384405
Tags:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20211024-181158382384405
Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk:20211024-181158382384405].
Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.7_sdk@sha256:633bcbbedad2b914103eba237fd407963850dd42fa52679f6ff8bb48f638b7de].
gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk  --format="get(digest)"
Digests:
- us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:e12279e9acbdef9b8813f5df9680bf57f7c39f40009b170066eaec64cc572d0c
  Associated tags:
 - 7c5442a1-f2be-4370-85bb-29de1100d4d9
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk:7c5442a1-f2be-4370-85bb-29de1100d4d9].
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python37_sdk/beam_python_prebuilt_sdk@sha256:e12279e9acbdef9b8813f5df9680bf57f7c39f40009b170066eaec64cc572d0c].
Removed the container

> Task :sdks:python:test-suites:dataflow:py37:validatesContainer FAILED

> Task :sdks:python:test-suites:dataflow:py36:validatesContainer
cleanup_container
docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH
Digests:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:c1ff0088b09c605f48013a558c199010c0c5ee9a7f9d733a9f585d1e398b786d
  Associated tags:
 - 20211024-181158734526802
Tags:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20211024-181158734526802
Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk:20211024-181158734526802].
Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.6_sdk@sha256:c1ff0088b09c605f48013a558c199010c0c5ee9a7f9d733a9f585d1e398b786d].
gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk  --format="get(digest)"
Digests:
- us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:ff725166d24e5417e7931975546532e16e4650497a5fdd683d015d2a529da9e9
  Associated tags:
 - b39c148a-9b93-4814-8fe3-c48f99199690
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk:b39c148a-9b93-4814-8fe3-c48f99199690].
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python36_sdk/beam_python_prebuilt_sdk@sha256:ff725166d24e5417e7931975546532e16e4650497a5fdd683d015d2a529da9e9].

> Task :sdks:python:test-suites:dataflow:py36:validatesContainer FAILED

> Task :sdks:python:test-suites:dataflow:py38:validatesContainer
cleanup_container
docker images --format '{{.Repository}}:{{.Tag}}' | grep $PREBUILD_SDK_CONTAINER_REGISTRY_PATH
Digests:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:abf25505f0c5522d39764eaeada9abcefd165b1b7cca1daaf23ad95b16fbd96b
  Associated tags:
 - 20211024-181204947611805
Tags:
- us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20211024-181204947611805
Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk:20211024-181204947611805].
Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python3.8_sdk@sha256:abf25505f0c5522d39764eaeada9abcefd165b1b7cca1daaf23ad95b16fbd96b].
gcloud container images list-tags $PREBUILD_SDK_CONTAINER_REGISTRY_PATH/beam_python_prebuilt_sdk  --format="get(digest)"
Digests:
- us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:cac7f0233fdae4d7f599da1073a7017ee845610456eb9c7682e45fbd23fc62f4
  Associated tags:
 - 9ad6fe8a-395c-4bb4-9652-ab2d8babcc45
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk:9ad6fe8a-395c-4bb4-9652-ab2d8babcc45].
Deleted [us.gcr.io/apache-beam-testing/jenkins/prebuild_python38_sdk/beam_python_prebuilt_sdk@sha256:cac7f0233fdae4d7f599da1073a7017ee845610456eb9c7682e45fbd23fc62f4].

> Task :sdks:python:test-suites:dataflow:py38:validatesContainer FAILED

FAILURE: Build completed with 3 failures.

1: Task failed with an exception.
-----------
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 276

* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:validatesContainer'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 276

* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py36:validatesContainer'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

3: Task failed with an exception.
-----------
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 276

* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py38:validatesContainer'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 36m 23s
44 actionable tasks: 40 executed, 4 up-to-date

Publishing build scan...
https://gradle.com/s/hi6mms7rckyqk

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org