You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2021/08/07 06:34:39 UTC
Build failed in Jenkins: beam_PostCommit_XVR_Dataflow #1042
See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/1042/display/redirect?page=changes>
Changes:
[noreply] Merge pull request #15246 from [BEAM-12685] Allow managed thread count
------------------------------------------
[...truncated 679.52 KB...]
self.result.wait_until_finish(duration=wait_duration)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <DataflowPipelineResult <Job
createTime: '2021-08-07T06:24:30.497805Z'
currentState: CurrentStateValueValuesEnum(JOB...021-08-07T06:24:30.497805Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> at 0x7fef3ee8a9e8>
duration = None
def wait_until_finish(self, duration=None):
if not self.is_in_terminal_state():
if not self.has_job:
raise IOError('Failed to get the Dataflow job id.')
thread = threading.Thread(
target=DataflowRunner.poll_for_job_completion,
args=(self._runner, self, duration))
# Mark the thread as a daemon thread so a keyboard interrupt on the main
# thread will terminate everything. This is also the reason we will not
# use thread.join() to wait for the polling thread.
thread.daemon = True
thread.start()
while thread.is_alive():
time.sleep(5.0)
# TODO: Merge the termination code in poll_for_job_completion and
# is_in_terminal_state.
terminated = self.is_in_terminal_state()
assert duration or terminated, (
'Job did not reach to a terminal state after waiting indefinitely.')
if terminated and self.state != PipelineState.DONE:
# TODO(BEAM-1290): Consider converting this to an error log based on
# theresolution of the issue.
raise DataflowRuntimeException(
'Dataflow pipeline failed. State: %s, Error:\n%s' %
(self.state, getattr(self._runner, 'last_error_msg', None)),
> self)
E apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
E Workflow failed. Causes: Job appears to be stuck. Several workers have failed to start up in a row, and no worker has successfully started up for this job. Last error reported: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210807060106 not found: manifest unknown: Failed to fetch "20210807060106" from request "/v2/apache-beam-testing/java-postcommit-it/java/manifests/20210807060106".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image..
apache_beam/runners/dataflow/dataflow_runner.py:1635: DataflowRuntimeException
------------------------------ Captured log call -------------------------------
INFO apache_beam.runners.portability.stager:stager.py:300 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz"> to staging location.
WARNING root:environments.py:374 Make sure that locally built Python SDK docker image has Python 3.6 interpreter.
INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.6_sdk:2.33.0.dev
INFO root:environments.py:296 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20210804
INFO root:environments.py:304 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python36-fnapi:beam-master-20210804" for Docker environment
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function pack_combiners at 0x7fef40caaf28> ====================
INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function sort_stages at 0x7fef40cac730> ====================
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:452 Defaulting to the temp_location as staging_location: gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/beam-sdks-java-testing-expansion-service-testExpansionService-2.33.0-SNAPSHOT-fhJoHz2J2dM-oxF9cVsN7RSsm3ykWyH458h6kvg3xPI.jar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/beam-sdks-java-testing-expansion-service-testExpansionService-2.33.0-SNAPSHOT-fhJoHz2J2dM-oxF9cVsN7RSsm3ykWyH458h6kvg3xPI.jar in 4 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/dataflow_python_sdk.tar...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/dataflow_python_sdk.tar in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:632 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/pipeline.pb...
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:651 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0807062424-032387.1628317464.033233/pipeline.pb in 0 seconds.
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:798 Create job: <Job
createTime: '2021-08-07T06:24:30.497805Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2021-08-06_23_24_29-5669045160880590923'
location: 'us-central1'
name: 'beamapp-jenkins-0807062424-032387'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2021-08-07T06:24:30.497805Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:800 Created job with id: [2021-08-06_23_24_29-5669045160880590923]
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:801 Submitted job: 2021-08-06_23_24_29-5669045160880590923
INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:807 To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-08-06_23_24_29-5669045160880590923?project=apache-beam-testing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-08-06_23_24_29-5669045160880590923 is in state JOB_STATE_RUNNING
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:33.162Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2021-08-06_23_24_29-5669045160880590923. The number of workers will be between 1 and 1000.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:33.248Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2021-08-06_23_24_29-5669045160880590923.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:35.631Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.250Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.271Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.336Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.369Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert_that/Group/_CoGBKImpl/GroupByKey: GroupByKey not followed by a combiner.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.394Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.414Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.443Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.476Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_assert_that-Group-_CoGBKImpl-Flatten_15
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.506Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_assert_that-Group-_CoGBKImpl-Flatten_15
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.552Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-_CoGBKImpl-Flatten_15 for input ref_AppliedPTransform_assert_that-Group-_CoGBKImpl-Tag-0-_13.None-post13
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.583Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/_CoGBKImpl/GroupByKey/Write, through flatten assert_that/Group/_CoGBKImpl/Flatten, into producer assert_that/Group/_CoGBKImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.616Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/MapTuple(collect_values) into assert_that/Group/_CoGBKImpl/GroupByKey/Read
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.648Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/RestoreTags into assert_that/Group/_CoGBKImpl/MapTuple(collect_values)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.682Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Unkey into assert_that/Group/RestoreTags
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.738Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Match into assert_that/Unkey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.776Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/GroupByKey/Write into assert_that/Group/_CoGBKImpl/Flatten/InputIdentity
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.808Z: JOB_MESSAGE_DETAILED: Fusing consumer GenerateSequence(beam:external:java:generate_sequence:v1)/Read(BoundedCountingSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into GenerateSequence(beam:external:java:generate_sequence:v1)/Read(BoundedCountingSource)/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.841Z: JOB_MESSAGE_DETAILED: Fusing consumer external_1GenerateSequence-beam-external-java-generate_sequence-v1--Read-BoundedCountingSource--ParDo-BoundedS/PairWithRestriction into GenerateSequence(beam:external:java:generate_sequence:v1)/Read(BoundedCountingSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.873Z: JOB_MESSAGE_DETAILED: Fusing consumer external_1GenerateSequence-beam-external-java-generate_sequence-v1--Read-BoundedCountingSource--ParDo-BoundedS/SplitWithSizing into external_1GenerateSequence-beam-external-java-generate_sequence-v1--Read-BoundedCountingSource--ParDo-BoundedS/PairWithRestriction
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.907Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/WindowInto(WindowIntoFn) into external_1GenerateSequence-beam-external-java-generate_sequence-v1--Read-BoundedCountingSource--ParDo-BoundedS/ProcessElementAndRestrictionWithSizing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.941Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/FlatMap(<lambda at core.py:2968>) into assert_that/Create/Impulse
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:36.998Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/Map(decode) into assert_that/Create/FlatMap(<lambda at core.py:2968>)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.031Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/Tag[0] into assert_that/Create/Map(decode)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.055Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/Flatten/InputIdentity into assert_that/Group/_CoGBKImpl/Tag[0]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.090Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/ToVoidKey into assert_that/WindowInto(WindowIntoFn)
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.122Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/Tag[1] into assert_that/ToVoidKey
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.145Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/Flatten/InputIdentity into assert_that/Group/_CoGBKImpl/Tag[1]
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.215Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.252Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.276Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.307Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.463Z: JOB_MESSAGE_DEBUG: Executing wait step start23
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.528Z: JOB_MESSAGE_BASIC: Executing operation GenerateSequence(beam:external:java:generate_sequence:v1)/Read(BoundedCountingSource)/Impulse+GenerateSequence(beam:external:java:generate_sequence:v1)/Read(BoundedCountingSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+external_1GenerateSequence-beam-external-java-generate_sequence-v1--Read-BoundedCountingSource--ParDo-BoundedS/PairWithRestriction+external_1GenerateSequence-beam-external-java-generate_sequence-v1--Read-BoundedCountingSource--ParDo-BoundedS/SplitWithSizing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.563Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/_CoGBKImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.576Z: JOB_MESSAGE_DEBUG: Starting worker pool setup.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.604Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-a...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.661Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/_CoGBKImpl/GroupByKey/Create
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.732Z: JOB_MESSAGE_DEBUG: Value "assert_that/Group/_CoGBKImpl/GroupByKey/Session" materialized.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:24:37.790Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:2968>)+assert_that/Create/Map(decode)+assert_that/Group/_CoGBKImpl/Tag[0]+assert_that/Group/_CoGBKImpl/Flatten/InputIdentity+assert_that/Group/_CoGBKImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:25:13.473Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:25:17.497Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:26:02.681Z: JOB_MESSAGE_ERROR: A worker was unable to start up. Error: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210807060106 not found: manifest unknown: Failed to fetch "20210807060106" from request "/v2/apache-beam-testing/java-postcommit-it/java/manifests/20210807060106".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:26:30.427Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:27:15.789Z: JOB_MESSAGE_ERROR: A worker was unable to start up. Error: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210807060106 not found: manifest unknown: Failed to fetch "20210807060106" from request "/v2/apache-beam-testing/java-postcommit-it/java/manifests/20210807060106".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:27:43.327Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:28:27.222Z: JOB_MESSAGE_ERROR: A worker was unable to start up. Error: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210807060106 not found: manifest unknown: Failed to fetch "20210807060106" from request "/v2/apache-beam-testing/java-postcommit-it/java/manifests/20210807060106".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:28:52.483Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:29:36.864Z: JOB_MESSAGE_ERROR: A worker was unable to start up. Error: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210807060106 not found: manifest unknown: Failed to fetch "20210807060106" from request "/v2/apache-beam-testing/java-postcommit-it/java/manifests/20210807060106".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:30:00.125Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s).
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:30:44.402Z: JOB_MESSAGE_ERROR: A worker was unable to start up. Error: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210807060106 not found: manifest unknown: Failed to fetch "20210807060106" from request "/v2/apache-beam-testing/java-postcommit-it/java/manifests/20210807060106".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:30:44.442Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: Job appears to be stuck. Several workers have failed to start up in a row, and no worker has successfully started up for this job. Last error reported: Unable to pull container image due to error: image pull request failed with error: Error response from daemon: manifest for us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210807060106 not found: manifest unknown: Failed to fetch "20210807060106" from request "/v2/apache-beam-testing/java-postcommit-it/java/manifests/20210807060106".. This is likely due to an invalid SDK container image URL. Please verify any provided SDK container image is valid and that Dataflow workers have permissions to pull image..
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:30:44.507Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:2968>)+assert_that/Create/Map(decode)+assert_that/Group/_CoGBKImpl/Tag[0]+assert_that/Group/_CoGBKImpl/Flatten/InputIdentity+assert_that/Group/_CoGBKImpl/GroupByKey/Write
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:30:44.893Z: JOB_MESSAGE_WARNING: S03:GenerateSequence(beam:external:java:generate_sequence:v1)/Read(BoundedCountingSource)/Impulse+GenerateSequence(beam:external:java:generate_sequence:v1)/Read(BoundedCountingSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+external_1GenerateSequence-beam-external-java-generate_sequence-v1--Read-BoundedCountingSource--ParDo-BoundedS/PairWithRestriction+external_1GenerateSequence-beam-external-java-generate_sequence-v1--Read-BoundedCountingSource--ParDo-BoundedS/SplitWithSizing failed.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:30:44.954Z: JOB_MESSAGE_BASIC: Finished operation GenerateSequence(beam:external:java:generate_sequence:v1)/Read(BoundedCountingSource)/Impulse+GenerateSequence(beam:external:java:generate_sequence:v1)/Read(BoundedCountingSource)/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)+external_1GenerateSequence-beam-external-java-generate_sequence-v1--Read-BoundedCountingSource--ParDo-BoundedS/PairWithRestriction+external_1GenerateSequence-beam-external-java-generate_sequence-v1--Read-BoundedCountingSource--ParDo-BoundedS/SplitWithSizing
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:30:45.057Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:30:45.125Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:30:45.183Z: JOB_MESSAGE_BASIC: Stopping worker pool...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:30:47.439Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:33:43.351Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-08-07T06:33:43.404Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-08-06_23_24_29-5669045160880590923 is in state JOB_STATE_FAILED
=============================== warnings summary ===============================
apache_beam/io/filesystems_test.py:54
apache_beam/io/filesystems_test.py:54
apache_beam/io/filesystems_test.py:54
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c
self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string
apache_beam/io/filesystems_test.py:62
apache_beam/io/filesystems_test.py:62
apache_beam/io/filesystems_test.py:62
<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d
self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/pytest_xlangValidateRunner.xml> -
=============== 7 failed, 1 passed, 6 warnings in 646.97 seconds ===============
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava FAILED
> Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup
Stopping expansion service pid: 7667.
Stopping expansion service pid: 7670.
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210807060106
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:718b32f091f3b79851079d807f8b2c4f41d800f7e9352b321c288d657e21398c
Deleted: sha256:e12d87f2939e6f37cb2a636125b0d2d828acd9f730097474254584f23e0e4bfd
Deleted: sha256:a7c4db42294155f392d88c60b4f5ebe78d1c6fb14fc331cfd72ca135b775574c
Deleted: sha256:6b0c6aa49fa260bedca7f8ea8065f064a8794a988f78fd5722f1f86b1f66331d
Deleted: sha256:37bfb4bb7be789754ae6a71fb9de89c05fca33f0e27ceed18f79a93ef4e8d5f3
Deleted: sha256:51c9f75791f7c4f9551ca09014f61c45f7a08d453e7e9f1ea0181d59a3a4e1bf
Deleted: sha256:2486f6308dcc9a0ef7196aa180e266b3253032a4b7ad56c84e34a84aaedd8a4b
Deleted: sha256:32ec60e495dff6df8535ac2373ad17cea004321310ca2f370e038809a8a2cbfa
Deleted: sha256:0b6900cfc71577484162c4bdb5c2ef66c8dc92c81c2bf5b56dd5d9f6014ce2f9
Deleted: sha256:112c864baa0448f2b6f2dbb00c2de7b94926a9078db9729adebbe1fdbaa97ce8
Deleted: sha256:7cd32482b2f26f08d7333ba80aea433c65b2aa25958afed6d112f28a7c6ca331
Deleted: sha256:79f33237d248c9c2a07a05def88823e4a2e4ff0a205f5ac7477c3d900c80d2f7
Deleted: sha256:b9d468fac573a39887e6ef171cc6726f0e6df0cc20ccb5a2ffb45ea78101797a
Deleted: sha256:38312fca139a0ed0b6ee83865957a02cdcb9704343f168badab1d4d1c3652c14
Deleted: sha256:6a22818b18efb9670d8cbd9f9c9164a5aea52e84a1f44df0b091418422d79599
Deleted: sha256:04df0df75bc5123efce760c38b8911da8538894b5073c86192c5d00e0946cba2
Deleted: sha256:c5f2ee0f6bb171cc24049f8082547fa0f915639f10f8e34505f1765dcffe9be6
Deleted: sha256:48c476f456221a7e3a7bad10f4f71b4c8ab2f0fcce0c7a08bfc849336f7ef544
Deleted: sha256:11fbf01c4ad0fd7cee193ee14009034d76f4499a604e1d0cb9e9d62cc0c74de0
Deleted: sha256:0d0b4cb7a905956ea0d2b7a80bd557872454e04bf51f39e7cd2bf06764ec09bd
Deleted: sha256:63f495373bfb319309e1b0c66aa3dd3e6a9c5f1819f402e6558e230eea4eb984
Deleted: sha256:d094541eb1c3644fc46adfa3dbcbd1632c5f0657fbb522752f8e9149edd97286
Deleted: sha256:0c85fd408b41678175171e34044de3da00f4057e82b9d5474859e89dc9cd2eae
Deleted: sha256:61610571c377b472a4d388ff29db34227f8e1aa45ab1889de93a770480db250c
Deleted: sha256:f1a1df8df812f9df5b86c56c8ab6773bcee80b45a76f57e5976a3a2a92028a97
Deleted: sha256:e632e5202279571e34f2061277d8782fcc2d3f1aecfe39810916bcfcdcb83c6c
Deleted: sha256:e3adad1b9d2f694c64ac816c7ac789e936e9fdbeb1b394e472ec058b02943bf2
Deleted: sha256:89a6ef30e15b33fb74faf488c3e69c8bc3b8d1a77f8beab62f41ef3595cb6f57
Deleted: sha256:606f8d30e7f634643b2aba7788f47d652a85256ce81bee910e5a9f6539799932
ERROR: (gcloud.container.images.untag) Image could not be found: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210807060106]
> Task :runners:google-cloud-dataflow-java:cleanUpDockerImages FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Build file '<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/runners/google-cloud-dataflow-java/build.gradle'> line: 279
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:cleanUpDockerImages'.
> Process 'command 'gcloud'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 34m 7s
113 actionable tasks: 96 executed, 13 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/uapnkgjsmiaz4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_XVR_Dataflow
#1043
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/1043/display/redirect>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org