You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2021/09/10 12:39:16 UTC
Build failed in Jenkins: beam_PostCommit_XVR_Dataflow #1179
See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/1179/display/redirect?page=changes>
Changes:
[noreply] Register MapCoder, some comments/cleanup. (#15471)
[noreply] [BEAM-12588] Multimap user state proto changes (#15473)
------------------------------------------
[...truncated 716.81 KB...]
apache_beam/testing/test_pipeline.py:112: in run
result = super(TestPipeline, self).run(
apache_beam/pipeline.py:566: in run
return self.runner.run_pipeline(self, self._options)
apache_beam/runners/dataflow/test_dataflow_runner.py:65: in run_pipeline
self.result.wait_until_finish(duration=wait_duration)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <DataflowPipelineResult <Job
clientRequestId: '20210910123712938350-9101'
createTime: '2021-09-10T12:37:19.073479Z'
...021-09-10T12:37:19.073479Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> at 0x7f0817706970>
duration = None
def wait_until_finish(self, duration=None):
if not self.is_in_terminal_state():
if not self.has_job:
raise IOError('Failed to get the Dataflow job id.')
thread = threading.Thread(
target=DataflowRunner.poll_for_job_completion,
args=(self._runner, self, duration))
# Mark the thread as a daemon thread so a keyboard interrupt on the main
# thread will terminate everything. This is also the reason we will not
# use thread.join() to wait for the polling thread.
thread.daemon = True
thread.start()
while thread.is_alive():
time.sleep(5.0)
# TODO: Merge the termination code in poll_for_job_completion and
# is_in_terminal_state.
terminated = self.is_in_terminal_state()
assert duration or terminated, (
'Job did not reach to a terminal state after waiting indefinitely.')
if terminated and self.state != PipelineState.DONE:
# TODO(BEAM-1290): Consider converting this to an error log based on
# theresolution of the issue.
> raise DataflowRuntimeException(
'Dataflow pipeline failed. State: %s, Error:\n%s' %
(self.state, getattr(self._runner, 'last_error_msg', None)),
self)
E apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
E Workflow failed.
apache_beam/runners/dataflow/dataflow_runner.py:1632: DataflowRuntimeException
------------------------------ Captured log call -------------------------------
INFO apache_beam.runners.portability.stager:stager.py:300 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING root:environments.py:371 Make sure that locally built Python SDK docker image has Python 3.8 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.8_sdk:2.34.0.dev
INFO root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python38-fnapi:beam-master-20210809
INFO root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python38-fnapi:beam-master-20210809" for Docker environment
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function pack_combiners at 0x7f07dc956e50> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function sort_stages at 0x7f07dc95a670> ====================
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:452 Defaulting to the temp_location as staging_location: gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/beam-sdks-java-testing-expansion-service-testExpansionService-2.34.0-SNAPSHOT-NGlHzUmCP1vJTQrCqB14AyQN4YDNio0H_h1Aw6NuE2A.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/beam-sdks-java-testing-expansion-service-testExpansionService-2.34.0-SNAPSHOT-NGlHzUmCP1vJTQrCqB14AyQN4YDNio0H_h1Aw6NuE2A.jar in 3 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/dataflow_python_sdk.tar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/dataflow_python_sdk.tar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/pipeline.pb...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0910123712-936151.1631277432.936764/pipeline.pb in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:819 Create job: <Job
clientRequestId: '20210910123712938350-9101'
createTime: '2021-09-10T12:37:19.073479Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-09-10_05_37_18-17202532670630972457'
location: 'us-central1'
name: 'beamapp-jenkins-0910123712-936151'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-09-10T12:37:19.073479Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:821 Created job with id: [2021-09-10_05_37_18-17202532670630972457]
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:822 Submitted job: 2021-09-10_05_37_18-17202532670630972457
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:823 To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-10_05_37_18-17202532670630972457?project=apache-beam-testing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-09-10_05_37_18-17202532670630972457 is in state JOB_STATE_RUNNING
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:21.849Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2021-09-10_05_37_18-17202532670630972457. The number of workers will be between 1 and 1000.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:21.924Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2021-09-10_05_37_18-17202532670630972457.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:23.594Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.163Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.191Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.255Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.289Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step check_odd/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.334Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step check_even/Group/CoGroupByKeyImpl/GroupByKey: GroupByKey not followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.360Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.391Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.422Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.471Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.492Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_check_even-Group-CoGroupByKeyImpl-Flatten_27
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.523Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_check_even-Group-CoGroupByKeyImpl-Flatten_27
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.556Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_check_odd-Group-CoGroupByKeyImpl-Flatten_45
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.586Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_check_odd-Group-CoGroupByKeyImpl-Flatten_45
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.660Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_check_even-Group-CoGroupByKeyImpl-Flatten_27 for input ref_AppliedPTransform_check_even-Group-CoGroupByKeyImpl-Tag-0-_25.None-post21
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.714Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of check_even/Group/CoGroupByKeyImpl/GroupByKey/Write, through flatten check_even/Group/CoGroupByKeyImpl/Flatten, into producer check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.756Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Group/CoGroupByKeyImpl/MapTuple(collect_values) into check_even/Group/CoGroupByKeyImpl/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.789Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Group/RestoreTags into check_even/Group/CoGroupByKeyImpl/MapTuple(collect_values)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.826Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Unkey into check_even/Group/RestoreTags
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.847Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Match into check_even/Unkey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.880Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_check_odd-Group-CoGroupByKeyImpl-Flatten_45 for input ref_AppliedPTransform_check_odd-Group-CoGroupByKeyImpl-Tag-0-_43.None-post23
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.909Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of check_odd/Group/CoGroupByKeyImpl/GroupByKey/Write, through flatten check_odd/Group/CoGroupByKeyImpl/Flatten, into producer check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.943Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Group/CoGroupByKeyImpl/MapTuple(collect_values) into check_odd/Group/CoGroupByKeyImpl/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:24.968Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Group/RestoreTags into check_odd/Group/CoGroupByKeyImpl/MapTuple(collect_values)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.001Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Unkey into check_odd/Group/RestoreTags
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.036Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Match into check_odd/Unkey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.068Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Group/CoGroupByKeyImpl/GroupByKey/Write into check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.105Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Group/CoGroupByKeyImpl/GroupByKey/Write into check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.137Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/FlatMap(<lambda at core.py:2965>) into Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.171Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/AddRandomKeys into Create/FlatMap(<lambda at core.py:2965>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.204Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create/MaybeReshuffle/Reshuffle/AddRandomKeys
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.238Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.273Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.323Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.351Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.385Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.407Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/Map(decode) into Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.433Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:partition)/ParMultiDo(Partition) into Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.462Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/WindowInto(WindowIntoFn) into ExternalTransform(beam:transforms:xlang:test:partition)/ParMultiDo(Partition)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.496Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/WindowInto(WindowIntoFn) into ExternalTransform(beam:transforms:xlang:test:partition)/ParMultiDo(Partition)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.527Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Create/FlatMap(<lambda at core.py:2965>) into check_even/Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.551Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Create/Map(decode) into check_even/Create/FlatMap(<lambda at core.py:2965>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.584Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Group/CoGroupByKeyImpl/Tag[0] into check_even/Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.640Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity into check_even/Group/CoGroupByKeyImpl/Tag[0]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.662Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/ToVoidKey into check_even/WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.713Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Group/CoGroupByKeyImpl/Tag[1] into check_even/ToVoidKey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.761Z: JOB_MESSAGE_DETAILED: Fusing consumer check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity into check_even/Group/CoGroupByKeyImpl/Tag[1]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.791Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Create/FlatMap(<lambda at core.py:2965>) into check_odd/Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.826Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Create/Map(decode) into check_odd/Create/FlatMap(<lambda at core.py:2965>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.861Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Group/CoGroupByKeyImpl/Tag[0] into check_odd/Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.884Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity into check_odd/Group/CoGroupByKeyImpl/Tag[0]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.918Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/ToVoidKey into check_odd/WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.948Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Group/CoGroupByKeyImpl/Tag[1] into check_odd/ToVoidKey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:25.980Z: JOB_MESSAGE_DETAILED: Fusing consumer check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity into check_odd/Group/CoGroupByKeyImpl/Tag[1]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.065Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.126Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.187Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.223Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.374Z: JOB_MESSAGE_DEBUG: Executing wait step start39
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.439Z: JOB_MESSAGE_BASIC: Executing operation Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.469Z: JOB_MESSAGE_BASIC: Executing operation check_even/Group/CoGroupByKeyImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.483Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.495Z: JOB_MESSAGE_BASIC: Executing operation check_odd/Group/CoGroupByKeyImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.520Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-a...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.554Z: JOB_MESSAGE_BASIC: Finished operation Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.568Z: JOB_MESSAGE_BASIC: Finished operation check_even/Group/CoGroupByKeyImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.568Z: JOB_MESSAGE_BASIC: Finished operation check_odd/Group/CoGroupByKeyImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.617Z: JOB_MESSAGE_DEBUG: Value "check_odd/Group/CoGroupByKeyImpl/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.658Z: JOB_MESSAGE_DEBUG: Value "check_even/Group/CoGroupByKeyImpl/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.707Z: JOB_MESSAGE_DEBUG: Value "Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.750Z: JOB_MESSAGE_BASIC: Executing operation check_odd/Create/Impulse+check_odd/Create/FlatMap(<lambda at core.py:2965>)+check_odd/Create/Map(decode)+check_odd/Group/CoGroupByKeyImpl/Tag[0]+check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity+check_odd/Group/CoGroupByKeyImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.772Z: JOB_MESSAGE_BASIC: Executing operation check_even/Create/Impulse+check_even/Create/FlatMap(<lambda at core.py:2965>)+check_even/Create/Map(decode)+check_even/Group/CoGroupByKeyImpl/Tag[0]+check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity+check_even/Group/CoGroupByKeyImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:26.804Z: JOB_MESSAGE_BASIC: Executing operation Create/Impulse+Create/FlatMap(<lambda at core.py:2965>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:37:46.308Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:38:06.763Z: JOB_MESSAGE_ERROR: Startup of the worker pool in zone us-central1-a failed to bring up any of the desired 1 workers. QUOTA_EXCEEDED: Quota 'CPUS' exceeded. Limit: 2500.0 in region us-central1.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:38:06.846Z: JOB_MESSAGE_ERROR: Workflow failed.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:38:06.991Z: JOB_MESSAGE_BASIC: Finished operation check_odd/Create/Impulse+check_odd/Create/FlatMap(<lambda at core.py:2965>)+check_odd/Create/Map(decode)+check_odd/Group/CoGroupByKeyImpl/Tag[0]+check_odd/Group/CoGroupByKeyImpl/Flatten/InputIdentity+check_odd/Group/CoGroupByKeyImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:38:06.991Z: JOB_MESSAGE_BASIC: Finished operation check_even/Create/Impulse+check_even/Create/FlatMap(<lambda at core.py:2965>)+check_even/Create/Map(decode)+check_even/Group/CoGroupByKeyImpl/Tag[0]+check_even/Group/CoGroupByKeyImpl/Flatten/InputIdentity+check_even/Group/CoGroupByKeyImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:38:06.992Z: JOB_MESSAGE_BASIC: Finished operation Create/Impulse+Create/FlatMap(<lambda at core.py:2965>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:38:07.070Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:38:07.128Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:38:07.160Z: JOB_MESSAGE_BASIC: Stopping worker pool...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:38:30.289Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-10T12:38:30.331Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-09-10_05_37_18-17202532670630972457 is in state JOB_STATE_FAILED
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def call(self, fn, *args, **kwargs):
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/pytest_xlangValidateRunner.xml> -
=================== 8 failed, 12 warnings in 142.68 seconds ====================
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava FAILED
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup
Stopping expansion service pid: 13287.
Stopping expansion service pid: 13290.
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210910122937
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8bebbee3f42be793b7b13acac87bc8b6e1628ed23984f1f9287e266d0ebaaceb
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210910122937]
- referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8bebbee3f42be793b7b13acac87bc8b6e1628ed23984f1f9287e266d0ebaaceb]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210910122937] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8bebbee3f42be793b7b13acac87bc8b6e1628ed23984f1f9287e266d0ebaaceb])].
Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8bebbee3f42be793b7b13acac87bc8b6e1628ed23984f1f9287e266d0ebaaceb
Digests:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8bebbee3f42be793b7b13acac87bc8b6e1628ed23984f1f9287e266d0ebaaceb
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:8bebbee3f42be793b7b13acac87bc8b6e1628ed23984f1f9287e266d0ebaaceb].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 39s
95 actionable tasks: 23 executed, 72 up-to-date
Publishing build scan...
https://gradle.com/s/zhdakch3dhtv2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_XVR_Dataflow
#1180
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/1180/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org