You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2021/07/23 12:59:06 UTC

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #8175

See <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/8175/display/redirect?page=changes>

Changes:

[noreply] [BEAM-12640] Enable adding JHM to Java based projects and add an example


------------------------------------------
[...truncated 1.18 MB...]
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125852-039787.1627045132.039952/dataflow-worker.jar in 5 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125852-039787.1627045132.039952/pipeline.pb...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125852-039787.1627045132.039952/pipeline.pb in 0 seconds.
WARNING  apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
WARNING  apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
______________ ReshuffleTest.test_reshuffle_preserves_timestamps _______________
[gw2] linux -- Python 3.8.5 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/bin/python3.8>

self = <apache_beam.transforms.util_test.ReshuffleTest testMethod=test_reshuffle_preserves_timestamps>

    @pytest.mark.it_validatesrunner
    def test_reshuffle_preserves_timestamps(self):
      with TestPipeline() as pipeline:
    
        # Create a PCollection and assign each element with a different timestamp.
        before_reshuffle = (
            pipeline
            | beam.Create([
                {
                    'name': 'foo', 'timestamp': MIN_TIMESTAMP
                },
                {
                    'name': 'foo', 'timestamp': 0
                },
                {
                    'name': 'bar', 'timestamp': 33
                },
                {
                    'name': 'bar', 'timestamp': 0
                },
            ])
            | beam.Map(
                lambda element: beam.window.TimestampedValue(
                    element, element['timestamp'])))
    
        # Reshuffle the PCollection above and assign the timestamp of an element
        # to that element again.
        after_reshuffle = before_reshuffle | beam.Reshuffle()
    
        # Given an element, emits a string which contains the timestamp and the
        # name field of the element.
        def format_with_timestamp(element, timestamp=beam.DoFn.TimestampParam):
          t = str(timestamp)
          if timestamp == MIN_TIMESTAMP:
            t = 'MIN_TIMESTAMP'
          elif timestamp == MAX_TIMESTAMP:
            t = 'MAX_TIMESTAMP'
          return '{} - {}'.format(t, element['name'])
    
        # Combine each element in before_reshuffle with its timestamp.
        formatted_before_reshuffle = (
            before_reshuffle
            | "Get before_reshuffle timestamp" >> beam.Map(format_with_timestamp))
    
        # Combine each element in after_reshuffle with its timestamp.
        formatted_after_reshuffle = (
            after_reshuffle
            | "Get after_reshuffle timestamp" >> beam.Map(format_with_timestamp))
    
        expected_data = [
            'MIN_TIMESTAMP - foo',
            'Timestamp(0) - foo',
            'Timestamp(33) - bar',
            'Timestamp(0) - bar'
        ]
    
        # Can't compare formatted_before_reshuffle and formatted_after_reshuffle
        # directly, because they are deferred PCollections while equal_to only
        # takes a concrete argument.
        assert_that(
            formatted_before_reshuffle,
            equal_to(expected_data),
            label="formatted_before_reshuffle")
>       assert_that(
            formatted_after_reshuffle,
            equal_to(expected_data),
            label="formatted_after_reshuffle")

apache_beam/transforms/util_test.py:611: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apache_beam/pipeline.py:585: in __exit__
    self.result = self.run()
apache_beam/testing/test_pipeline.py:112: in run
    result = super(TestPipeline, self).run(
apache_beam/pipeline.py:537: in run
    return Pipeline.from_runner_api(
apache_beam/pipeline.py:564: in run
    return self.runner.run_pipeline(self, self._options)
apache_beam/runners/dataflow/test_dataflow_runner.py:53: in run_pipeline
    self.result = super(TestDataflowRunner,
apache_beam/runners/dataflow/dataflow_runner.py:588: in run_pipeline
    self.dataflow_client.create_job(self.job), self)
apache_beam/utils/retry.py:253: in wrapper
    return fun(*args, **kwargs)
apache_beam/runners/dataflow/internal/apiclient.py:688: in create_job
    return self.submit_job_description(job)
apache_beam/utils/retry.py:253: in wrapper
    return fun(*args, **kwargs)
apache_beam/runners/dataflow/internal/apiclient.py:789: in submit_job_description
    response = self._client.projects_locations_jobs.Create(request)
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py:902: in Create
    return self._RunMethod(
../../build/gradleenv/-1734967051/lib/python3.8/site-packages/apitools/base/py/base_api.py:731: in _RunMethod
    return self.ProcessHttpResponse(method_config, http_response, request)
../../build/gradleenv/-1734967051/lib/python3.8/site-packages/apitools/base/py/base_api.py:737: in ProcessHttpResponse
    self.__ProcessHttpResponse(method_config, http_response, request))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3.ProjectsLocationsJobsService object at 0x7fc9685bc6d0>
method_config = <ApiMethodInfo
 relative_path: 'v1b3/projects/{projectId}/locations/{location}/jobs'
 method_id: 'dataflow.projects.lo...DataflowProjectsLocationsJobsCreateRequest'
 response_type_name: 'Job'
 request_field: 'job'
 supports_download: False>
http_response = Response(info={'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Fri, 2...', request_url='https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json')
request = <DataflowProjectsLocationsJobsCreateRequest
 job: <Job
 environment: <Environment
 clusterManagerApiService: 'compute....iles: []
 type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
 location: 'us-central1'
 projectId: 'apache-beam-testing'>

    def __ProcessHttpResponse(self, method_config, http_response, request):
        """Process the given http response."""
        if http_response.status_code not in (http_client.OK,
                                             http_client.CREATED,
                                             http_client.NO_CONTENT):
>           raise exceptions.HttpError.FromResponse(
                http_response, method_config=method_config, request=request)
E           apitools.base.py.exceptions.HttpBadRequestError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Fri, 23 Jul 2021 12:59:02 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '400', 'content-length': '237', '-content-encoding': 'gzip'}>, content <{
E             "error": {
E               "code": 400,
E               "message": "(983af44b3a88e684): The workflow could not be created. Causes: (983af44b3a88eab3): A Cloud KMS key cannot be used with Streaming Engine enabled.",
E               "status": "FAILED_PRECONDITION"
E             }
E           }
E           >

../../build/gradleenv/-1734967051/lib/python3.8/site-packages/apitools/base/py/base_api.py:603: HttpBadRequestError
------------------------------ Captured log call -------------------------------
INFO     apache_beam.runners.portability.stager:stager.py:644 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/bin/python3.8',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', 'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']
INFO     apache_beam.runners.portability.stager:stager.py:300 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING  root:environments.py:371 Make sure that locally built Python SDK docker image has Python 3.8 interpreter.
INFO     root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.8_sdk:2.33.0.dev
INFO     root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python38-fnapi:beam-master-20210720
INFO     root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python38-fnapi:beam-master-20210720" for Docker environment
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/requirements.txt...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/requirements.txt in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/pbr-5.5.1.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/pbr-5.5.1.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/pbr-5.6.0.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/pbr-5.6.0.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/mock-2.0.0.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/mock-2.0.0.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/six-1.16.0.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/six-1.16.0.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/soupsieve-2.1.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/soupsieve-2.1.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/soupsieve-2.2.1.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/soupsieve-2.2.1.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/PyHamcrest-1.10.1.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/PyHamcrest-1.10.1.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/parameterized-0.7.5.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/parameterized-0.7.5.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/beautifulsoup4-4.9.3.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/beautifulsoup4-4.9.3.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/dataflow_python_sdk.tar...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/dataflow_python_sdk.tar in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/dataflow-worker.jar...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/dataflow-worker.jar in 5 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/pipeline.pb...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723125854-477331.1627045134.477492/pipeline.pb in 0 seconds.
WARNING  apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
WARNING  apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
=============================== warnings summary ===============================
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
  <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
    def call(self, fn, *args, **kwargs):

-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/pytest_validatesRunnerStreamingTests-df-py38-xdist.xml> -
========= 26 failed, 3 passed, 3 skipped, 8 warnings in 72.17 seconds ==========

> Task :sdks:python:test-suites:dataflow:py38:validatesRunnerStreamingTests FAILED

FAILURE: Build completed with 3 failures.

1: Task failed with an exception.
-----------
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 189

* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 189

* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py36:validatesRunnerStreamingTests'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

3: Task failed with an exception.
-----------
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 189

* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py38:validatesRunnerStreamingTests'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 58m 34s
92 actionable tasks: 82 executed, 10 from cache

Publishing build scan...
https://gradle.com/s/n4ttpb3jbpbbi

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org


Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #8177

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/8177/display/redirect?page=changes>


---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org


Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #8176

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/8176/display/redirect?page=changes>

Changes:

[noreply] [BEAM-12652] revving JS filename to bust cache (#15209)


------------------------------------------
[...truncated 1.18 MB...]
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184105-465851.1627065665.466022/dataflow-worker.jar in 5 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184105-465851.1627065665.466022/pipeline.pb...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184105-465851.1627065665.466022/pipeline.pb in 0 seconds.
WARNING  apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
WARNING  apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
______________ ReshuffleTest.test_reshuffle_preserves_timestamps _______________
[gw3] linux -- Python 3.8.5 <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/bin/python3.8>

self = <apache_beam.transforms.util_test.ReshuffleTest testMethod=test_reshuffle_preserves_timestamps>

    @pytest.mark.it_validatesrunner
    def test_reshuffle_preserves_timestamps(self):
      with TestPipeline() as pipeline:
    
        # Create a PCollection and assign each element with a different timestamp.
        before_reshuffle = (
            pipeline
            | beam.Create([
                {
                    'name': 'foo', 'timestamp': MIN_TIMESTAMP
                },
                {
                    'name': 'foo', 'timestamp': 0
                },
                {
                    'name': 'bar', 'timestamp': 33
                },
                {
                    'name': 'bar', 'timestamp': 0
                },
            ])
            | beam.Map(
                lambda element: beam.window.TimestampedValue(
                    element, element['timestamp'])))
    
        # Reshuffle the PCollection above and assign the timestamp of an element
        # to that element again.
        after_reshuffle = before_reshuffle | beam.Reshuffle()
    
        # Given an element, emits a string which contains the timestamp and the
        # name field of the element.
        def format_with_timestamp(element, timestamp=beam.DoFn.TimestampParam):
          t = str(timestamp)
          if timestamp == MIN_TIMESTAMP:
            t = 'MIN_TIMESTAMP'
          elif timestamp == MAX_TIMESTAMP:
            t = 'MAX_TIMESTAMP'
          return '{} - {}'.format(t, element['name'])
    
        # Combine each element in before_reshuffle with its timestamp.
        formatted_before_reshuffle = (
            before_reshuffle
            | "Get before_reshuffle timestamp" >> beam.Map(format_with_timestamp))
    
        # Combine each element in after_reshuffle with its timestamp.
        formatted_after_reshuffle = (
            after_reshuffle
            | "Get after_reshuffle timestamp" >> beam.Map(format_with_timestamp))
    
        expected_data = [
            'MIN_TIMESTAMP - foo',
            'Timestamp(0) - foo',
            'Timestamp(33) - bar',
            'Timestamp(0) - bar'
        ]
    
        # Can't compare formatted_before_reshuffle and formatted_after_reshuffle
        # directly, because they are deferred PCollections while equal_to only
        # takes a concrete argument.
        assert_that(
            formatted_before_reshuffle,
            equal_to(expected_data),
            label="formatted_before_reshuffle")
>       assert_that(
            formatted_after_reshuffle,
            equal_to(expected_data),
            label="formatted_after_reshuffle")

apache_beam/transforms/util_test.py:611: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apache_beam/pipeline.py:585: in __exit__
    self.result = self.run()
apache_beam/testing/test_pipeline.py:112: in run
    result = super(TestPipeline, self).run(
apache_beam/pipeline.py:537: in run
    return Pipeline.from_runner_api(
apache_beam/pipeline.py:564: in run
    return self.runner.run_pipeline(self, self._options)
apache_beam/runners/dataflow/test_dataflow_runner.py:53: in run_pipeline
    self.result = super(TestDataflowRunner,
apache_beam/runners/dataflow/dataflow_runner.py:588: in run_pipeline
    self.dataflow_client.create_job(self.job), self)
apache_beam/utils/retry.py:253: in wrapper
    return fun(*args, **kwargs)
apache_beam/runners/dataflow/internal/apiclient.py:688: in create_job
    return self.submit_job_description(job)
apache_beam/utils/retry.py:253: in wrapper
    return fun(*args, **kwargs)
apache_beam/runners/dataflow/internal/apiclient.py:789: in submit_job_description
    response = self._client.projects_locations_jobs.Create(request)
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py:902: in Create
    return self._RunMethod(
../../build/gradleenv/-1734967051/lib/python3.8/site-packages/apitools/base/py/base_api.py:731: in _RunMethod
    return self.ProcessHttpResponse(method_config, http_response, request)
../../build/gradleenv/-1734967051/lib/python3.8/site-packages/apitools/base/py/base_api.py:737: in ProcessHttpResponse
    self.__ProcessHttpResponse(method_config, http_response, request))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <apache_beam.runners.dataflow.internal.clients.dataflow.dataflow_v1b3_client.DataflowV1b3.ProjectsLocationsJobsService object at 0x7f818c6c1310>
method_config = <ApiMethodInfo
 relative_path: 'v1b3/projects/{projectId}/locations/{location}/jobs'
 method_id: 'dataflow.projects.lo...DataflowProjectsLocationsJobsCreateRequest'
 response_type_name: 'Job'
 request_field: 'job'
 supports_download: False>
http_response = Response(info={'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Fri, 2...', request_url='https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json')
request = <DataflowProjectsLocationsJobsCreateRequest
 job: <Job
 environment: <Environment
 clusterManagerApiService: 'compute....iles: []
 type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
 location: 'us-central1'
 projectId: 'apache-beam-testing'>

    def __ProcessHttpResponse(self, method_config, http_response, request):
        """Process the given http response."""
        if http_response.status_code not in (http_client.OK,
                                             http_client.CREATED,
                                             http_client.NO_CONTENT):
>           raise exceptions.HttpError.FromResponse(
                http_response, method_config=method_config, request=request)
E           apitools.base.py.exceptions.HttpBadRequestError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/apache-beam-testing/locations/us-central1/jobs?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Fri, 23 Jul 2021 18:41:13 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '400', 'content-length': '237', '-content-encoding': 'gzip'}>, content <{
E             "error": {
E               "code": 400,
E               "message": "(454047edae6738d1): The workflow could not be created. Causes: (454047edae673ccc): A Cloud KMS key cannot be used with Streaming Engine enabled.",
E               "status": "FAILED_PRECONDITION"
E             }
E           }
E           >

../../build/gradleenv/-1734967051/lib/python3.8/site-packages/apitools/base/py/base_api.py:603: HttpBadRequestError
------------------------------ Captured log call -------------------------------
INFO     apache_beam.runners.portability.stager:stager.py:644 Executing command: ['<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/bin/python3.8',> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', 'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']
INFO     apache_beam.runners.portability.stager:stager.py:300 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING  root:environments.py:371 Make sure that locally built Python SDK docker image has Python 3.8 interpreter.
INFO     root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.8_sdk:2.33.0.dev
INFO     root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python38-fnapi:beam-master-20210720
INFO     root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python38-fnapi:beam-master-20210720" for Docker environment
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/requirements.txt...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/requirements.txt in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/pbr-5.5.1.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/pbr-5.5.1.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/pbr-5.6.0.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/pbr-5.6.0.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/mock-2.0.0.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/mock-2.0.0.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/six-1.16.0.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/six-1.16.0.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/soupsieve-2.1.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/soupsieve-2.1.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/soupsieve-2.2.1.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/soupsieve-2.2.1.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/PyHamcrest-1.10.1.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/PyHamcrest-1.10.1.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/parameterized-0.7.5.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/parameterized-0.7.5.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/beautifulsoup4-4.9.3.tar.gz...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/beautifulsoup4-4.9.3.tar.gz in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/dataflow_python_sdk.tar...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/dataflow_python_sdk.tar in 0 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/dataflow-worker.jar...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/dataflow-worker.jar in 5 seconds.
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/pipeline.pb...
INFO     apache_beam.runners.dataflow.internal.apiclient:apiclient.py:648 Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0723184106-060510.1627065666.060682/pipeline.pb in 0 seconds.
WARNING  apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
WARNING  apache_beam.options.pipeline_options:pipeline_options.py:309 Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
=============================== warnings summary ===============================
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
  <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/build/gradleenv/-1734967051/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
    def call(self, fn, *args, **kwargs):

-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/pytest_validatesRunnerStreamingTests-df-py38-xdist.xml> -
========= 26 failed, 3 passed, 3 skipped, 8 warnings in 78.00 seconds ==========

> Task :sdks:python:test-suites:dataflow:py38:validatesRunnerStreamingTests FAILED

FAILURE: Build completed with 3 failures.

1: Task failed with an exception.
-----------
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 189

* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:validatesRunnerStreamingTests'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 189

* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py36:validatesRunnerStreamingTests'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

3: Task failed with an exception.
-----------
* Where:
Script '<https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/test-suites/dataflow/common.gradle'> line: 189

* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py38:validatesRunnerStreamingTests'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 40m 44s
92 actionable tasks: 62 executed, 30 from cache

Publishing build scan...
https://gradle.com/s/jgfqg3yk4xcqk

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org