You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2021/06/25 13:16:06 UTC
Build failed in Jenkins: beam_PostCommit_XVR_Dataflow #873
See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/873/display/redirect>
Changes:
------------------------------------------
[...truncated 1.14 MB...]
# Mark the thread as a daemon thread so a keyboard interrupt on the main
# thread will terminate everything. This is also the reason we will not
# use thread.join() to wait for the polling thread.
thread.daemon = True
thread.start()
while thread.is_alive():
time.sleep(5.0)
# TODO: Merge the termination code in poll_for_job_completion and
# is_in_terminal_state.
terminated = self.is_in_terminal_state()
assert duration or terminated, (
'Job did not reach to a terminal state after waiting indefinitely.')
if terminated and self.state != PipelineState.DONE:
# TODO(BEAM-1290): Consider converting this to an error log based on
# theresolution of the issue.
raise DataflowRuntimeException(
'Dataflow pipeline failed. State: %s, Error:\n%s' %
(self.state, getattr(self._runner, 'last_error_msg', None)),
> self)
E apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
E The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. Please check the worker logs in Stackdriver Logging. You can also get help with Cloud Dataflow at https://cloud.google.com/dataflow/support.
apache_beam/runners/dataflow/dataflow_runner.py:1629: DataflowRuntimeException
------------------------------ Captured log call -------------------------------
INFO apache_beam.runners.portability.stager:stager.py:300 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.6 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.6_sdk:2.32.0.dev
INFO root:environments.py:296 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20210622
INFO root:environments.py:304 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20210622" for Docker environment
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function pack_combiners at 0x7f3224c0dd90> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function sort_stages at 0x7f3224c0b598> ====================
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:452 Defaulting to the temp_location as staging_location: gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/beam-sdks-java-testing-expansion-service-testExpansionService-2.32.0-SNAPSHOT-r2gF5NoKrv2Y2yzSA5y9TNf74qdrZxzSVFwgQYg5t7M.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/beam-sdks-java-testing-expansion-service-testExpansionService-2.32.0-SNAPSHOT-r2gF5NoKrv2Y2yzSA5y9TNf74qdrZxzSVFwgQYg5t7M.jar in 7 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/dataflow_python_sdk.tar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/dataflow_python_sdk.tar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/pipeline.pb...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625121437-655259.1624623277.656063/pipeline.pb in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:798 Create job: <Job
createTime: '2021-06-25T12:14:47.815942Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-25_05_14_46-11711171666216555371'
location: 'us-central1'
name: 'beamapp-jenkins-0625121437-655259'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-25T12:14:47.815942Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:800 Created job with id: [2021-06-25_05_14_46-11711171666216555371]
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:801 Submitted job: 2021-06-25_05_14_46-11711171666216555371
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:807 To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-25_05_14_46-11711171666216555371?project=apache-beam-testing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-06-25_05_14_46-11711171666216555371 is in state JOB_STATE_RUNNING
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:50.043Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2021-06-25_05_14_46-11711171666216555371. The number of workers will be between 1 and 1000.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:50.237Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2021-06-25_05_14_46-11711171666216555371.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:52.505Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.066Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.092Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.151Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.187Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.231Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.302Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.350Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.396Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.426Z: JOB_MESSAGE_DEBUG: Inserted coder converter after flatten ref_AppliedPTransform_assert_that-Group-Flatten_26
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.459Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-Flatten_26 for input ref_AppliedPTransform_assert_that-Group-pair_with_0_24.None
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.491Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/Flatten/OutputIdentity, through flatten assert_that/Group/Flatten, into producer assert_that/Group/pair_with_0
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.523Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/Map(_merge_tagged_vals_under_key) into assert_that/Group/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.561Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.595Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Match into assert_that/Unkey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.630Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-Flatten_26-u40 for input ref_AppliedPTransform_assert_that-Group-Flatten_26.None-c38
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.715Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/GroupByKey/Write, through flatten assert_that/Group/Flatten/Unzipped-1, into producer assert_that/Group/Flatten/OutputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.741Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/Flatten/OutputIdentity into assert_that/Group/pair_with_1
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.772Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/GroupByKey/Write into assert_that/Group/Flatten/OutputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.803Z: JOB_MESSAGE_DETAILED: Unzipping flatten external_1ExternalTransform-beam-transforms-xlang-test-comgl--Flatten-PCollections for input external_1ExternalTransform-beam-transforms-xlang-test-comgl--ProduceDefault-ParMultiDo-Anonymous-.output
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.834Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/WindowInto(WindowIntoFn), through flatten ExternalTransform(beam:transforms:xlang:test:comgl)/Flatten.PCollections, into producer ExternalTransform(beam:transforms:xlang:test:comgl)/ProduceDefault/ParMultiDo(Anonymous)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.894Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/pair_with_1 into assert_that/ToVoidKey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.932Z: JOB_MESSAGE_DETAILED: Unzipping flatten external_1ExternalTransform-beam-transforms-xlang-test-u47 for input ref_AppliedPTransform_assert_that-WindowInto-WindowIntoFn-_21.None-c45
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.965Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/ToVoidKey, through flatten ExternalTransform(beam:transforms:xlang:test:comgl)/Flatten.PCollections/Unzipped-1, into producer assert_that/WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:53.995Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:comgl)/View.AsIterable/MapElements/Map/ParMultiDo(Anonymous) into ExternalTransform(beam:transforms:xlang:test:comgl)/Values/Values/Map/ParMultiDo(Anonymous)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.019Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/WindowInto(WindowIntoFn) into ExternalTransform(beam:transforms:xlang:test:comgl)/Values/Values/Map/ParMultiDo(Anonymous)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.064Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/ToVoidKey into assert_that/WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.089Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/FlatMap(<lambda at core.py:2972>) into Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.120Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/AddRandomKeys into Create/FlatMap(<lambda at core.py:2972>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.152Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create/MaybeReshuffle/Reshuffle/AddRandomKeys
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.189Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.220Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.249Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.290Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.330Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.370Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/Map(decode) into Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.400Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:comgl)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous) into Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.426Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey+ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues/Partial into ExternalTransform(beam:transforms:xlang:test:comgl)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.457Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey/Write into ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey+ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues/Partial
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.501Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues into ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.538Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues/Extract into ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.578Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:comgl)/Values/Values/Map/ParMultiDo(Anonymous) into ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/Combine.GroupedValues/Extract
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.616Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.648Z: JOB_MESSAGE_DETAILED: Fusing consumer external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/PairWithRestriction into ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.715Z: JOB_MESSAGE_DETAILED: Fusing consumer external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/SplitWithSizing into external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/PairWithRestriction
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.747Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:comgl)/ProduceDefault/ParMultiDo(Anonymous) into external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/ProcessElementAndRestrictionWithSizing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.783Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/FlatMap(<lambda at core.py:2972>) into assert_that/Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.817Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/Map(decode) into assert_that/Create/FlatMap(<lambda at core.py:2972>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.846Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/pair_with_0 into assert_that/Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.880Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.912Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.938Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:54.970Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.206Z: JOB_MESSAGE_DEBUG: Executing wait step start65
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.311Z: JOB_MESSAGE_BASIC: Executing operation ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/Impulse+ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/PairWithRestriction+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/SplitWithSizing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.342Z: JOB_MESSAGE_BASIC: Executing operation ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.356Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.383Z: JOB_MESSAGE_BASIC: Executing operation Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.383Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-f...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.418Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.444Z: JOB_MESSAGE_BASIC: Finished operation ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.444Z: JOB_MESSAGE_BASIC: Finished operation Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.460Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.506Z: JOB_MESSAGE_DEBUG: Value "ExternalTransform(beam:transforms:xlang:test:comgl)/Combine.perKey(SumLong)/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.533Z: JOB_MESSAGE_DEBUG: Value "Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.562Z: JOB_MESSAGE_DEBUG: Value "assert_that/Group/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.600Z: JOB_MESSAGE_BASIC: Executing operation Create/Impulse+Create/FlatMap(<lambda at core.py:2972>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:14:55.633Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:2972>)+assert_that/Create/Map(decode)+assert_that/Group/pair_with_0+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:15:27.354Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:15:35.799Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:15:52.190Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T12:15:52.218Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO oauth2client.transport:transport.py:185 Refreshing due to a 401 (attempt 1/2)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:14:55.755Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. Please check the worker logs in Stackdriver Logging. You can also get help with Cloud Dataflow at https://cloud.google.com/dataflow/support.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:14:55.837Z: JOB_MESSAGE_BASIC: Cancel request is committed for workflow job: 2021-06-25_05_14_46-11711171666216555371.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:14:55.858Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:2972>)+assert_that/Create/Map(decode)+assert_that/Group/pair_with_0+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:14:55.858Z: JOB_MESSAGE_BASIC: Finished operation Create/Impulse+Create/FlatMap(<lambda at core.py:2972>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:14:55.961Z: JOB_MESSAGE_ERROR: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. Please check the worker logs in Stackdriver Logging. You can also get help with Cloud Dataflow at https://cloud.google.com/dataflow/support.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:14:56.176Z: JOB_MESSAGE_WARNING: S11:ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/Impulse+ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/PairWithRestriction+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/SplitWithSizing failed.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:14:56.232Z: JOB_MESSAGE_BASIC: Finished operation ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/Impulse+ExternalTransform(beam:transforms:xlang:test:comgl)/CreateVoid/Read(CreateSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/PairWithRestriction+external_1ExternalTransform-beam-transforms-xlang-test-comgl--CreateVoid-Read-CreateSource--ParDo-BoundedSourc/SplitWithSizing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:14:56.302Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:14:56.359Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:14:56.386Z: JOB_MESSAGE_BASIC: Stopping worker pool...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:15:46.519Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:15:46.558Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T13:15:46.619Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-06-25_05_14_46-11711171666216555371 is in state JOB_STATE_FAILED
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/pytest_xlangValidateRunner.xml> -
============== 7 failed, 1 passed, 2 warnings in 3715.54 seconds ===============
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava FAILED
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup
Stopping expansion service pid: 9100.
Stopping expansion service pid: 9103.
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages FAILED
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210625120053
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:1b8055650dd42418c63e6c375a669dc2a1e12518251b48cda07449fb5c18a6c1
Deleted: sha256:df396190e9cdbc4646fbac6a2a43f761126b351719f61e89b2b269aaa1d20f65
Deleted: sha256:42ae504540be985bf7205700fd5794f0b3b309646cba9975b605f44a101c3975
Deleted: sha256:0ab4a75a59a33775efa6c0f549b34ef1a5f73ae8311dad3178e97070c68e946f
Deleted: sha256:fff736185224eddfa8fdf7355694ebc88e762865959c827898656e69a3bf3d3b
Deleted: sha256:26b4c2696ef09db3724a2fed44548e894c0e03fd50bab4084485fd069ff2dc25
Deleted: sha256:276815d9822c904ec2fa491314ea5b19585fce176e7cee4690e740978df820b0
Deleted: sha256:035c3e442ed6ea535ac76b7d005c7365b463d81d9819b91a80a9c0971acdb079
Deleted: sha256:06b02657b5f2d0dd07bc0759b0aae41fbaaebbe1a35552752a9258907d4c0a5f
Deleted: sha256:73854191cc9a488a02a0a049b6d57406f7eedc9c81ad363137de29ef100d69e7
Deleted: sha256:112c0a73437033325f5d76e68da67c3cabfa44f8a767ffd7bf24569386ada064
Deleted: sha256:082acc4819890a74ed8e28111eb9f6fc7842e02540b05954ab40f5dbf0f3c7cb
Deleted: sha256:c779ac1a23ae6de111c2ffc2f5636500ec73d53bcc79fcd3aac548698561f84a
Deleted: sha256:63fa8ce6fc5457eeaee02a4fe296dddeaa43990e862699be966acc5ebb729779
Deleted: sha256:492e8cc6dfe071423d4959845413600eb831cf4ab8087ef61607e8156f232f95
Deleted: sha256:a8c0d5f43ad09447381277fcbeca8d59709af8388c8eb23cce16a0a002b41493
Deleted: sha256:d1ca16b311fe11172c1e60078b6ecc81c1e2412be10dac2421f66b9760316cad
Deleted: sha256:b630c27b83c7f9426147b9262a7d42cdb9624132831fbece64450dc2c0302344
Deleted: sha256:c0a4f8444a7a7a7ac640701efc215e236d18dd339546e737e0a70484ba823a11
Deleted: sha256:f0e0eaca6139f53105f5adb3fb4e17c60fe89a56f25d8660d811e3e67caab7be
Deleted: sha256:4a3b33071e4089f2d8a12a67eb7fddddd7be33b4d2bef9d0af80780b3c6d2d4f
Deleted: sha256:44c4e09fab17f1ce621ef4ca1b59c2183488ecad3747ffd4f1a31f10c6eacf0e
Deleted: sha256:be954a8d80baddfb8d3ca658ae018d99a3fbece9d74011cf1990c57ddb35d8b4
Deleted: sha256:d58a58635a69dd8423cdc38d2ea26e7bd99f7ba1612550bcca43d5312353d519
Deleted: sha256:c76470090b2177863520ba98a220aaf272daff149ebfff2a883761858732d782
Deleted: sha256:1284e3df8a0acd36a8a872d268f1b2a3eb86f0ecdd48d36b1fb740654537cbe7
Deleted: sha256:7d617ba0944c373965cc364797ef496449640a25dd6f58c035abef7d63adca6b
Deleted: sha256:b9d77802d10d59512f6706309ccfd4e24a48d39382811a1133350c4f31f482f2
Deleted: sha256:fb4a7d30708e44f165330328ca1b7dd408af5e2f63dacc7373d4e7f3678c20de
ERROR: (gcloud.container.images.delete) [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210625120053] is not a valid name. Expected tag in the form "base:tag" or "tag" or digest in the form "sha256:<digest>"
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/runners/google-cloud-dataflow-java/build.gradle'> line: 279
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:cleanUpDockerImages'.
> Process 'command 'gcloud'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 15m 37s
113 actionable tasks: 84 executed, 25 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/c72tr2zjaqfiy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_XVR_Dataflow #875
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/875/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_XVR_Dataflow #874
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/874/display/redirect?page=changes>
Changes:
[noreply] [BEAM-9547] Add implementation for from_dict, from_records (#15034)
[noreply] [BEAM-11951] added "Differences from Pandas" page for DataFrame (#15074)
[noreply] [BEAM-9547] reindex is order-sensitive (#15032)
------------------------------------------
[...truncated 1.23 MB...]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:51.396Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:51.438Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-06-25_11_31_29-12476500062621846372 is in state JOB_STATE_FAILED
________ ValidateRunnerXlangTest.test_multi_input_output_with_sideinput ________
[gw3] linux -- Python 3.6.8 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/bin/python3.6>
self = <apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest testMethod=test_multi_input_output_with_sideinput>
test_pipeline = None
@pytest.mark.xlang_transforms
def test_multi_input_output_with_sideinput(self, test_pipeline=None):
CrossLanguageTestPipelines().run_multi_input_output_with_sideinput(
> test_pipeline or self.create_pipeline())
apache_beam/transforms/validate_runner_xlang_test.py:259:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/transforms/validate_runner_xlang_test.py:121: in run_multi_input_output_with_sideinput
assert_that(res['side'], equal_to(['ss']), label='CheckSide')
apache_beam/pipeline.py:585: in __exit__
self.result = self.run()
apache_beam/testing/test_pipeline.py:114: in run
False if self.not_use_test_runner_api else test_runner_api))
apache_beam/pipeline.py:564: in run
return self.runner.run_pipeline(self, self._options)
apache_beam/runners/dataflow/test_dataflow_runner.py:65: in run_pipeline
self.result.wait_until_finish(duration=wait_duration)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <DataflowPipelineResult <Job
createTime: '2021-06-25T18:31:31.008704Z'
currentState: CurrentStateValueValuesEnum(JOB...021-06-25T18:31:31.008704Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> at 0x7f96f9195fd0>
duration = None
def wait_until_finish(self, duration=None):
if not self.is_in_terminal_state():
if not self.has_job:
raise IOError('Failed to get the Dataflow job id.')
thread = threading.Thread(
target=DataflowRunner.poll_for_job_completion,
args=(self._runner, self, duration))
# Mark the thread as a daemon thread so a keyboard interrupt on the main
# thread will terminate everything. This is also the reason we will not
# use thread.join() to wait for the polling thread.
thread.daemon = True
thread.start()
while thread.is_alive():
time.sleep(5.0)
# TODO: Merge the termination code in poll_for_job_completion and
# is_in_terminal_state.
terminated = self.is_in_terminal_state()
assert duration or terminated, (
'Job did not reach to a terminal state after waiting indefinitely.')
if terminated and self.state != PipelineState.DONE:
# TODO(BEAM-1290): Consider converting this to an error log based on
# theresolution of the issue.
raise DataflowRuntimeException(
'Dataflow pipeline failed. State: %s, Error:\n%s' %
(self.state, getattr(self._runner, 'last_error_msg', None)),
> self)
E apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
E Workflow failed.
apache_beam/runners/dataflow/dataflow_runner.py:1629: DataflowRuntimeException
------------------------------ Captured log call -------------------------------
INFO apache_beam.runners.portability.stager:stager.py:300 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.6 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.6_sdk:2.32.0.dev
INFO root:environments.py:296 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20210622
INFO root:environments.py:304 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20210622" for Docker environment
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function pack_combiners at 0x7f96fb43b2f0> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function sort_stages at 0x7f96fb43ba60> ====================
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:452 Defaulting to the temp_location as staging_location: gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/beam-sdks-java-testing-expansion-service-testExpansionService-2.32.0-SNAPSHOT-r2gF5NoKrv2Y2yzSA5y9TNf74qdrZxzSVFwgQYg5t7M.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/beam-sdks-java-testing-expansion-service-testExpansionService-2.32.0-SNAPSHOT-r2gF5NoKrv2Y2yzSA5y9TNf74qdrZxzSVFwgQYg5t7M.jar in 5 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/dataflow_python_sdk.tar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/dataflow_python_sdk.tar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/pipeline.pb...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0625183122-142941.1624645882.143741/pipeline.pb in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:798 Create job: <Job
createTime: '2021-06-25T18:31:31.008704Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-06-25_11_31_29-1061865775003673254'
location: 'us-central1'
name: 'beamapp-jenkins-0625183122-142941'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-06-25T18:31:31.008704Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:800 Created job with id: [2021-06-25_11_31_29-1061865775003673254]
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:801 Submitted job: 2021-06-25_11_31_29-1061865775003673254
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:807 To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-06-25_11_31_29-1061865775003673254?project=apache-beam-testing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-06-25_11_31_29-1061865775003673254 is in state JOB_STATE_RUNNING
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:33.251Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2021-06-25_11_31_29-1061865775003673254. The number of workers will be between 1 and 1000.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:33.398Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2021-06-25_11_31_29-1061865775003673254.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:35.667Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-b.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.224Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.263Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.339Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.374Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.404Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step CheckSide/Group/GroupByKey: GroupByKey not followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.497Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.536Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.588Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.625Z: JOB_MESSAGE_DEBUG: Inserted coder converter after flatten ref_AppliedPTransform_CheckSide-Group-Flatten_45
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.657Z: JOB_MESSAGE_DEBUG: Inserted coder converter after flatten ref_AppliedPTransform_assert_that-Group-Flatten_29
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.684Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-Flatten_29 for input ref_AppliedPTransform_assert_that-Group-pair_with_0_27.None
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.719Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/Flatten/OutputIdentity, through flatten assert_that/Group/Flatten, into producer assert_that/Group/pair_with_0
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.741Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/Map(_merge_tagged_vals_under_key) into assert_that/Group/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.773Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.810Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Match into assert_that/Unkey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.844Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-Flatten_29-u32 for input ref_AppliedPTransform_assert_that-Group-Flatten_29.None-c30
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.877Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/GroupByKey/Write, through flatten assert_that/Group/Flatten/Unzipped-1, into producer assert_that/Group/Flatten/OutputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.917Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/Flatten/OutputIdentity into assert_that/Group/pair_with_1
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.956Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/GroupByKey/Write into assert_that/Group/Flatten/OutputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:36.987Z: JOB_MESSAGE_DETAILED: Unzipping flatten external_1ExternalTransform-beam-transforms-xlang-test-multi--Flatten-PCollections for input ref_AppliedPTransform_Main1-Map-decode-_6.None
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.027Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of ExternalTransform(beam:transforms:xlang:test:multi)/ParDo(Anonymous)/ParMultiDo(Anonymous), through flatten ExternalTransform(beam:transforms:xlang:test:multi)/Flatten.PCollections, into producer Main1/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.059Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_CheckSide-Group-Flatten_45 for input ref_AppliedPTransform_CheckSide-Group-pair_with_0_43.None
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.107Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of CheckSide/Group/Flatten/OutputIdentity, through flatten CheckSide/Group/Flatten, into producer CheckSide/Group/pair_with_0
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.142Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/Group/Map(_merge_tagged_vals_under_key) into CheckSide/Group/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.176Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/Unkey into CheckSide/Group/Map(_merge_tagged_vals_under_key)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.198Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/Match into CheckSide/Unkey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.226Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/ToVoidKey into assert_that/WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.262Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/pair_with_1 into assert_that/ToVoidKey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.294Z: JOB_MESSAGE_DETAILED: Unzipping flatten external_1ExternalTransform-beam-transforms-xlang-test-multi--Flatten-PCollections-u39 for input external_1ExternalTransform-beam-transforms-xlang-test-multi--ParDo-Anonymous--ParMultiDo-Anonymous-.output-c37
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.328Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/WindowInto(WindowIntoFn), through flatten ExternalTransform(beam:transforms:xlang:test:multi)/Flatten.PCollections/Unzipped-1, into producer ExternalTransform(beam:transforms:xlang:test:multi)/ParDo(Anonymous)/ParMultiDo(Anonymous)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.364Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_CheckSide-Group-Flatten_45-u44 for input ref_AppliedPTransform_CheckSide-Group-Flatten_45.None-c42
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.395Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of CheckSide/Group/GroupByKey/Write, through flatten CheckSide/Group/Flatten/Unzipped-1, into producer CheckSide/Group/Flatten/OutputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.447Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:multi)/MapElements/Map/ParMultiDo(Anonymous) into Side/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.483Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/WithKeys/AddKeys/Map/ParMultiDo(Anonymous) into Side/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.504Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:multi)/ParDo(Anonymous)/ParMultiDo(Anonymous) into Main2/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.537Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/Group/Flatten/OutputIdentity into CheckSide/Group/pair_with_1
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.575Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/WindowInto(WindowIntoFn) into ExternalTransform(beam:transforms:xlang:test:multi)/ParDo(Anonymous)/ParMultiDo(Anonymous)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.610Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/Group/GroupByKey/Write into CheckSide/Group/Flatten/OutputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.643Z: JOB_MESSAGE_DETAILED: Fusing consumer Side/FlatMap(<lambda at core.py:2972>) into Side/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.676Z: JOB_MESSAGE_DETAILED: Fusing consumer Side/Map(decode) into Side/FlatMap(<lambda at core.py:2972>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.697Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey+ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/Combine.GroupedValues/Partial into ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.728Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey/Write into ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey+ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.757Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/Combine.GroupedValues into ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.786Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/Combine.GroupedValues/Extract into ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/Combine.GroupedValues
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.817Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Values/Values/Map/ParMultiDo(Anonymous) into ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.851Z: JOB_MESSAGE_DETAILED: Fusing consumer Main1/FlatMap(<lambda at core.py:2972>) into Main1/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.886Z: JOB_MESSAGE_DETAILED: Fusing consumer Main1/Map(decode) into Main1/FlatMap(<lambda at core.py:2972>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.920Z: JOB_MESSAGE_DETAILED: Fusing consumer Main2/FlatMap(<lambda at core.py:2972>) into Main2/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.945Z: JOB_MESSAGE_DETAILED: Fusing consumer Main2/Map(decode) into Main2/FlatMap(<lambda at core.py:2972>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:37.977Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/WindowInto(WindowIntoFn) into ExternalTransform(beam:transforms:xlang:test:multi)/MapElements/Map/ParMultiDo(Anonymous)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.007Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/Create/FlatMap(<lambda at core.py:2972>) into CheckSide/Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.041Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/Create/Map(decode) into CheckSide/Create/FlatMap(<lambda at core.py:2972>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.065Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/Group/pair_with_0 into CheckSide/Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.111Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/ToVoidKey into CheckSide/WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.137Z: JOB_MESSAGE_DETAILED: Fusing consumer CheckSide/Group/pair_with_1 into CheckSide/ToVoidKey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.161Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/FlatMap(<lambda at core.py:2972>) into assert_that/Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.200Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/Map(decode) into assert_that/Create/FlatMap(<lambda at core.py:2972>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.232Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/pair_with_0 into assert_that/Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.272Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.309Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.343Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.375Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.537Z: JOB_MESSAGE_DEBUG: Executing wait step start67
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.606Z: JOB_MESSAGE_BASIC: Executing operation ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.640Z: JOB_MESSAGE_BASIC: Executing operation CheckSide/Group/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.654Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.673Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.683Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-b...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.727Z: JOB_MESSAGE_BASIC: Finished operation ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.739Z: JOB_MESSAGE_BASIC: Finished operation CheckSide/Group/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.739Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.800Z: JOB_MESSAGE_DEBUG: Value "ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.827Z: JOB_MESSAGE_DEBUG: Value "assert_that/Group/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.855Z: JOB_MESSAGE_DEBUG: Value "CheckSide/Group/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.884Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:2972>)+assert_that/Create/Map(decode)+assert_that/Group/pair_with_0+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.945Z: JOB_MESSAGE_BASIC: Executing operation Side/Impulse+Side/FlatMap(<lambda at core.py:2972>)+Side/Map(decode)+ExternalTransform(beam:transforms:xlang:test:multi)/MapElements/Map/ParMultiDo(Anonymous)+ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)+ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey+ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey/Write+CheckSide/WindowInto(WindowIntoFn)+CheckSide/ToVoidKey+CheckSide/Group/pair_with_1+CheckSide/Group/Flatten/OutputIdentity+CheckSide/Group/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:38.977Z: JOB_MESSAGE_BASIC: Executing operation CheckSide/Create/Impulse+CheckSide/Create/FlatMap(<lambda at core.py:2972>)+CheckSide/Create/Map(decode)+CheckSide/Group/pair_with_0+CheckSide/Group/Flatten/OutputIdentity+CheckSide/Group/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:31:58.941Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:27.064Z: JOB_MESSAGE_ERROR: Startup of the worker pool in zone us-central1-b failed to bring up any of the desired 1 workers. ZONE_RESOURCE_POOL_EXHAUSTED: The zone 'projects/apache-beam-testing/zones/us-central1-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:27.094Z: JOB_MESSAGE_ERROR: Workflow failed.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:27.165Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:2972>)+assert_that/Create/Map(decode)+assert_that/Group/pair_with_0+assert_that/Group/Flatten/OutputIdentity+assert_that/Group/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:27.165Z: JOB_MESSAGE_BASIC: Finished operation Side/Impulse+Side/FlatMap(<lambda at core.py:2972>)+Side/Map(decode)+ExternalTransform(beam:transforms:xlang:test:multi)/MapElements/Map/ParMultiDo(Anonymous)+ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)+ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey+ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/Combine.GroupedValues/Partial+ExternalTransform(beam:transforms:xlang:test:multi)/View.AsSingleton/Combine.GloballyAsSingletonView/CombineValues/Combine.perKey(Singleton)/GroupByKey/Write+CheckSide/WindowInto(WindowIntoFn)+CheckSide/ToVoidKey+CheckSide/Group/pair_with_1+CheckSide/Group/Flatten/OutputIdentity+CheckSide/Group/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:27.165Z: JOB_MESSAGE_BASIC: Finished operation CheckSide/Create/Impulse+CheckSide/Create/FlatMap(<lambda at core.py:2972>)+CheckSide/Create/Map(decode)+CheckSide/Group/pair_with_0+CheckSide/Group/Flatten/OutputIdentity+CheckSide/Group/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:27.244Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:27.315Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:27.342Z: JOB_MESSAGE_BASIC: Stopping worker pool...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:49.824Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-06-25T18:32:49.863Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-06-25_11_31_29-1061865775003673254 is in state JOB_STATE_FAILED
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/pytest_xlangValidateRunner.xml> -
========================== 8 failed in 131.65 seconds ==========================
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava FAILED
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup
Stopping expansion service pid: 26291.
Stopping expansion service pid: 26295.
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210625182329
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:e63bb731a6673bd8150ebcb90295ee4948933f82ab779aecaa619740601448f6
Digests:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:e63bb731a6673bd8150ebcb90295ee4948933f82ab779aecaa619740601448f6
Associated tags:
- 20210625182329
Tags:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210625182329
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210625182329].
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:e63bb731a6673bd8150ebcb90295ee4948933f82ab779aecaa619740601448f6].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 38s
103 actionable tasks: 29 executed, 74 up-to-date
Publishing build scan...
https://gradle.com/s/afmzdy36pqc2k
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org