You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2022/05/03 13:27:21 UTC
Build failed in Jenkins: beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17 #134
See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/134/display/redirect?page=changes>
Changes:
[yathu] [BEAM-14375] Fix Java Wordcount Dataflow postcommit
[Robert Bradshaw] Allow arithmetic between deferred scalars.
[noreply] [BEAM-14390] Set user-agent when pulling licenses to avoid 403s (#17521)
[noreply] [BEAM-8688] Upgrade GCSIO to 2.2.6 (#17486)
[noreply] [BEAM-14253] patch SubscriptionPartitionLoader to work around a dataflow
[noreply] Add website link log to notify user of pre-build workflow. (#17498)
[noreply] [BEAM-11105] Add timestamp observing watermark estimation (#17476)
[noreply] Merge pull request #17487 from Adding user-agent to GCS client in Python
[noreply] [BEAM-10265] Display error message if trying to infer recursive schema
[noreply] [BEAM-12575] Upgraded ipykernel from v5 to v6 (#17526)
[noreply] [BEAM-11105] Add docs + CHANGES.md entry for Go Watermark Estimation
[noreply] Merge pull request #17380 from [BEAM-14314][BEAM-9532] Add last_updated
------------------------------------------
[...truncated 45.93 KB...]
> Task :release:go-licenses:java:dockerRun
+ go-licenses save github.com/apache/beam/sdks/java/container --save_path=/output/licenses
+ go-licenses csv github.com/apache/beam/sdks/java/container
+ tee /output/licenses/list.csv
+ chmod -R a+w /output/licenses
> Task :release:go-licenses:java:createLicenses
> Task :sdks:java:container:java17:copyGolangLicenses
> Task :sdks:java:container:java17:dockerPrepare
> Task :sdks:java:container:java17:docker
> Task :runners:google-cloud-dataflow-java:buildAndPushDockerJavaContainer
WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03.
As an alternative, use `gcloud auth configure-docker` to configure `docker` to
use `gcloud` as a credential helper, then use `docker` as you would for non-GCR
registries, e.g. `docker pull gcr.io/project-id/my-image`. Add
`--verbosity=error` to silence this warning: `gcloud docker
--verbosity=error -- pull gcr.io/project-id/my-image`.
See: https://cloud.google.com/container-registry/docs/support/deprecation-notices#gcloud-docker
The push refers to repository [us.gcr.io/apache-beam-testing/java-postcommit-it/java]
52eb74fe55c7: Preparing
a0047307b411: Preparing
22f5c719ac2a: Preparing
571a60fbcd54: Preparing
5cafb396ea19: Preparing
b9bb22d55e3e: Preparing
cf1017aad468: Preparing
d825907d653d: Preparing
8db3937692b0: Preparing
bac86cd751e4: Preparing
9fd422015e17: Preparing
5e79d87fff88: Preparing
51a379cd90f3: Preparing
ea4b0956d47c: Preparing
75795add637a: Preparing
3bc383470c05: Preparing
e93827457889: Preparing
08fa02ce37eb: Preparing
a037458de4e0: Preparing
bafdbe68e4ae: Preparing
a13c519c6361: Preparing
ea4b0956d47c: Waiting
75795add637a: Waiting
8db3937692b0: Waiting
a037458de4e0: Waiting
b9bb22d55e3e: Waiting
bac86cd751e4: Waiting
bafdbe68e4ae: Waiting
3bc383470c05: Waiting
a13c519c6361: Waiting
9fd422015e17: Waiting
51a379cd90f3: Waiting
e93827457889: Waiting
08fa02ce37eb: Waiting
d825907d653d: Waiting
cf1017aad468: Waiting
5cafb396ea19: Pushed
22f5c719ac2a: Pushed
a0047307b411: Pushed
571a60fbcd54: Pushed
52eb74fe55c7: Pushed
cf1017aad468: Pushed
d825907d653d: Pushed
bac86cd751e4: Pushed
8db3937692b0: Pushed
5e79d87fff88: Pushed
b9bb22d55e3e: Pushed
3bc383470c05: Layer already exists
e93827457889: Layer already exists
9fd422015e17: Pushed
08fa02ce37eb: Layer already exists
a037458de4e0: Layer already exists
bafdbe68e4ae: Layer already exists
a13c519c6361: Layer already exists
ea4b0956d47c: Pushed
75795add637a: Pushed
51a379cd90f3: Pushed
20220503123036: digest: sha256:d3e45bd5042d6ca189215f85538763fdb51a453a71b9f9ce4dd097d09e46a5dd size: 4729
> Task :sdks:java:testing:load-tests:run
May 03, 2022 12:32:36 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 03, 2022 12:32:36 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 221 files. Enable logging at DEBUG level to see which files will be staged.
May 03, 2022 12:32:37 PM org.apache.beam.sdk.Pipeline validate
WARNING: The following transforms do not have stable unique names: Window.Into()
May 03, 2022 12:32:37 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
May 03, 2022 12:32:41 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 221 files from PipelineOptions.filesToStage to staging location to prepare for execution.
May 03, 2022 12:32:43 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 221 files cached, 0 files newly uploaded in 2 seconds
May 03, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/
May 03, 2022 12:32:44 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <119943 bytes, hash 59f916c5e19489bd88ce5811761090b62724c332f6f3411ef2c783946ee7ef48> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-WfkWxeGUib2IzlgRdhCQtickwzL280Ee8seDlG7n70g.pb
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1
May 03, 2022 12:32:45 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@463561c5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@659feb22, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3468ee6e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2f4b98f6, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@421def93, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@58c1da09, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2b2954e1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@58d6e55a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@751ae8a4, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@235d659c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4232b34a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2da16263, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@f5ce0bb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@47e51549, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@101a461c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@360e9c06, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5ebffb44, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@311ff287, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7377781e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@31db34da]
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/StripIds as step s2
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (input) as step s3
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s4
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s5
May 03, 2022 12:32:45 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3a2e9f5b, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7da34b26, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@10c47c79, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3b2553d9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@753fd7a1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@12a2585b, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@21b6c9c2, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@309cedb6, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3b95a6db, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bf54172, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2c9a6717, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7b3cde6f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6d091cad, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7c663eaf, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bb5ceb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1e692555, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3ba0ae41, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@76fe6cdc, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2ffb3aec, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@786ff1cb]
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/StripIds as step s6
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (co-input) as step s7
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()2/Window.Assign as step s8
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable0 as step s9
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable1 as step s10
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/Flatten as step s11
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/GBK as step s12
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s13
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate as step s14
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect total bytes as step s15
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics as step s16
May 03, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.39.0-SNAPSHOT
May 03, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-03_05_32_46-11901399501120612900?project=apache-beam-testing
May 03, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-05-03_05_32_46-11901399501120612900
May 03, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2022-05-03_05_32_46-11901399501120612900
May 03, 2022 12:32:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-05-03T12:32:50.653Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java170dataflow0v20streaming0cogbk01-jenkins-05-trww. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions
May 03, 2022 12:32:58 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:58.016Z: Worker configuration: e2-standard-2 in us-central1-b.
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:58.605Z: Expanding SplittableParDo operations into optimizable parts.
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:58.682Z: Expanding CollectionToSingleton operations into optimizable parts.
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.174Z: Expanding CoGroupByKey operations into optimizable parts.
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.417Z: Expanding SplittableProcessKeyed operations into optimizable parts.
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.458Z: Expanding GroupByKey operations into streaming Read/Write steps
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.533Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.645Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.687Z: Unzipping flatten CoGroupByKey-Flatten for input CoGroupByKey-MakeUnionTable0-ParMultiDo-ConstructUnionTable-.output
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.721Z: Fusing unzipped copy of CoGroupByKey/GBK/WriteStream, through flatten CoGroupByKey/Flatten, into producer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable)
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.767Z: Fusing consumer CoGroupByKey/GBK/WriteStream into CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable)
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.798Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.831Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.865Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.897Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.929Z: Fusing consumer Collect start time metrics (input)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds)
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.964Z: Fusing consumer Window.Into()/Window.Assign into Collect start time metrics (input)/ParMultiDo(TimeMonitor)
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:32:59.995Z: Fusing consumer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) into Window.Into()/Window.Assign
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:00.049Z: Fusing consumer Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read co-input/Impulse
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:00.097Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:00.227Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 03, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:00.316Z: Fusing consumer Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 03, 2022 12:33:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:00.528Z: Fusing consumer Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) into Read co-input/ParDo(StripIds)/ParMultiDo(StripIds)
May 03, 2022 12:33:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:00.660Z: Fusing consumer Window.Into()2/Window.Assign into Collect start time metrics (co-input)/ParMultiDo(TimeMonitor)
May 03, 2022 12:33:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:00.811Z: Fusing consumer CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) into Window.Into()2/Window.Assign
May 03, 2022 12:33:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:00.895Z: Fusing consumer CoGroupByKey/GBK/MergeBuckets into CoGroupByKey/GBK/ReadStream
May 03, 2022 12:33:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:00.945Z: Fusing consumer CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) into CoGroupByKey/GBK/MergeBuckets
May 03, 2022 12:33:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:00.987Z: Fusing consumer Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) into CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult)
May 03, 2022 12:33:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:01.020Z: Fusing consumer Collect total bytes/ParMultiDo(ByteMonitor) into Ungroup and reiterate/ParMultiDo(UngroupAndReiterate)
May 03, 2022 12:33:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:01.061Z: Fusing consumer Collect end time metrics/ParMultiDo(TimeMonitor) into Collect total bytes/ParMultiDo(ByteMonitor)
May 03, 2022 12:33:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:01.275Z: Running job using Streaming Engine
May 03, 2022 12:33:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:01.528Z: Starting 5 ****s in us-central1-b...
May 03, 2022 12:33:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:18.790Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
May 03, 2022 12:33:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:33:24.290Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 03, 2022 12:34:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-03T12:34:36.212Z: Workers have started successfully.
FATAL: command execution failed
java.io.IOException: Backing channel 'apache-beam-jenkins-10' is disconnected.
at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:216)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
at com.sun.proxy.$Proxy132.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1213)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1205)
at hudson.Launcher$ProcStarter.join(Launcher.java:522)
at hudson.plugins.gradle.Gradle.perform(Gradle.java:317)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:806)
at hudson.model.Build$BuildExecution.build(Build.java:198)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:514)
at hudson.model.Run.execute(Run.java:1888)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:99)
at hudson.model.Executor.run(Executor.java:432)
Caused by: hudson.remoting.Channel$OrderlyShutdown: Command Close created at
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1320)
at hudson.remoting.Channel$1.handle(Channel.java:607)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:81)
Caused by: Command Close created at
at hudson.remoting.Command.<init>(Command.java:70)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1313)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1311)
at hudson.remoting.Channel.close(Channel.java:1487)
at hudson.remoting.Channel.close(Channel.java:1454)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1319)
... 2 more
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
ERROR: apache-beam-jenkins-10 is offline; cannot locate jdk_1.8_latest
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17 #142
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/142/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17 #141
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/141/display/redirect?page=changes>
Changes:
[andyye333] Add extra details to PubSub matcher errors
[noreply] Merge pull request #17559 from [BEAM-14423] Add exception injection
[noreply] [BEAM-11104] Allow self-checkpointing SDFs to return without finishing
[noreply] Merge pull request #17544 from [BEAM-14415] Exception handling tests for
[noreply] Merge pull request #17565 from [BEAM-14413] add Kafka exception test
[noreply] Merge pull request #17555 from [BEAM-14417] Adding exception handling
[noreply] [BEAM-14433] Improve Go split error message. (#17575)
[noreply] [BEAM-14429] Force java load test on dataflow runner v2
[noreply] Merge pull request #17577 from [BEAM-14435] Adding exception handling
[noreply] [BEAM-14347] Add generic registration functions for iters and emitters
[noreply] [BEAM-14169] Add Credentials rotation cron job for clusters (#17383)
[noreply] [BEAM-14347] Add generic registration for accumulators (#17579)
------------------------------------------
[...truncated 50.68 KB...]
8d3c4b8a056c: Preparing
779e60d40a60: Preparing
6e94b4ebcad2: Preparing
3a7cd99b8479: Preparing
c49c9f3cb678: Preparing
5a27e519e570: Waiting
3fe3663cf18e: Preparing
3634662e7534: Preparing
779e60d40a60: Waiting
6e94b4ebcad2: Waiting
267e49f181ed: Preparing
3bc383470c05: Preparing
e93827457889: Preparing
08fa02ce37eb: Preparing
a037458de4e0: Preparing
bafdbe68e4ae: Preparing
a13c519c6361: Preparing
08fa02ce37eb: Waiting
a13c519c6361: Waiting
a037458de4e0: Waiting
8d3c4b8a056c: Waiting
e93827457889: Waiting
3a7cd99b8479: Waiting
267e49f181ed: Waiting
c49c9f3cb678: Waiting
3bc383470c05: Waiting
83bac70d0466: Waiting
aa1c4990f94b: Pushed
5d41c137d1a8: Pushed
7d9293365344: Pushed
38be9522fc38: Pushed
a51548066d49: Pushed
8d3c4b8a056c: Pushed
5a27e519e570: Pushed
779e60d40a60: Pushed
6e94b4ebcad2: Pushed
c49c9f3cb678: Pushed
3a7cd99b8479: Pushed
3bc383470c05: Layer already exists
83bac70d0466: Pushed
e93827457889: Layer already exists
08fa02ce37eb: Layer already exists
a037458de4e0: Layer already exists
bafdbe68e4ae: Layer already exists
a13c519c6361: Layer already exists
3634662e7534: Pushed
267e49f181ed: Pushed
3fe3663cf18e: Pushed
20220510123038: digest: sha256:681c0d5f2a230de72485a61d99b98d34569d0303dda01320734c2ebe64c4a3da size: 4729
> Task :sdks:java:testing:load-tests:run
May 10, 2022 12:32:49 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 10, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 222 files. Enable logging at DEBUG level to see which files will be staged.
May 10, 2022 12:32:51 PM org.apache.beam.sdk.Pipeline validate
WARNING: The following transforms do not have stable unique names: Window.Into()
May 10, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
May 10, 2022 12:32:53 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 222 files from PipelineOptions.filesToStage to staging location to prepare for execution.
May 10, 2022 12:32:54 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 222 files cached, 0 files newly uploaded in 0 seconds
May 10, 2022 12:32:54 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/
May 10, 2022 12:32:54 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <120279 bytes, hash 90d5250ac2615b6f15499695641c4bbc2e25c31d4352ec686073b7925252605d> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-kNUlCsJhW28VSZaVZBxLvC4lwx1DUuxoYHO3klJSYF0.pb
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1
May 10, 2022 12:32:56 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 1 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4232b34a]
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/StripIds as step s2
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (input) as step s3
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s4
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s5
May 10, 2022 12:32:56 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 1 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@ffd4cba]
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/StripIds as step s6
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (co-input) as step s7
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()2/Window.Assign as step s8
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable0 as step s9
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable1 as step s10
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/Flatten as step s11
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/GBK as step s12
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s13
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate as step s14
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect total bytes as step s15
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics as step s16
May 10, 2022 12:32:56 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.40.0-SNAPSHOT
May 10, 2022 12:32:57 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-10_05_32_56-7371717862294852586?project=apache-beam-testing
May 10, 2022 12:32:57 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-05-10_05_32_56-7371717862294852586
May 10, 2022 12:32:57 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2022-05-10_05_32_56-7371717862294852586
May 10, 2022 12:33:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-05-10T12:33:02.262Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java170dataflow0v20streaming0cogbk01-jenkins-05-ahvo. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions
May 10, 2022 12:33:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:12.593Z: Worker configuration: e2-standard-2 in us-central1-b.
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:13.927Z: Expanding SplittableParDo operations into optimizable parts.
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:13.960Z: Expanding CollectionToSingleton operations into optimizable parts.
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.024Z: Expanding CoGroupByKey operations into optimizable parts.
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.093Z: Expanding SplittableProcessKeyed operations into optimizable parts.
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.125Z: Expanding GroupByKey operations into streaming Read/Write steps
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.186Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.308Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.336Z: Unzipping flatten CoGroupByKey-Flatten for input CoGroupByKey-MakeUnionTable0-ParMultiDo-ConstructUnionTable-.output
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.371Z: Fusing unzipped copy of CoGroupByKey/GBK/WriteStream, through flatten CoGroupByKey/Flatten, into producer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable)
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.405Z: Fusing consumer CoGroupByKey/GBK/WriteStream into CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable)
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.439Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.476Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.506Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.543Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.575Z: Fusing consumer Collect start time metrics (input)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds)
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.606Z: Fusing consumer Window.Into()/Window.Assign into Collect start time metrics (input)/ParMultiDo(TimeMonitor)
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.659Z: Fusing consumer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) into Window.Into()/Window.Assign
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.694Z: Fusing consumer Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read co-input/Impulse
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.738Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.772Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.797Z: Fusing consumer Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.827Z: Fusing consumer Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) into Read co-input/ParDo(StripIds)/ParMultiDo(StripIds)
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.864Z: Fusing consumer Window.Into()2/Window.Assign into Collect start time metrics (co-input)/ParMultiDo(TimeMonitor)
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.889Z: Fusing consumer CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) into Window.Into()2/Window.Assign
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.918Z: Fusing consumer CoGroupByKey/GBK/MergeBuckets into CoGroupByKey/GBK/ReadStream
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.952Z: Fusing consumer CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) into CoGroupByKey/GBK/MergeBuckets
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:14.987Z: Fusing consumer Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) into CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult)
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:15.020Z: Fusing consumer Collect total bytes/ParMultiDo(ByteMonitor) into Ungroup and reiterate/ParMultiDo(UngroupAndReiterate)
May 10, 2022 12:33:15 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:15.047Z: Fusing consumer Collect end time metrics/ParMultiDo(TimeMonitor) into Collect total bytes/ParMultiDo(ByteMonitor)
May 10, 2022 12:33:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:15.240Z: Running job using Streaming Engine
May 10, 2022 12:33:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:15.523Z: Starting 5 ****s in us-central1-b...
May 10, 2022 12:33:22 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:21.076Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
May 10, 2022 12:33:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:33:38.137Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 10, 2022 12:34:47 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T12:34:46.474Z: Workers have started successfully.
May 10, 2022 2:13:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T14:13:36.440Z: Cleaning up.
May 10, 2022 2:13:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T14:13:36.646Z: Stopping **** pool...
May 10, 2022 2:13:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T14:13:36.700Z: Stopping **** pool...
May 10, 2022 2:14:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T14:14:11.917Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
May 10, 2022 2:14:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-10T14:14:11.960Z: Worker pool stopped.
May 10, 2022 2:14:18 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2022-05-10_05_32_56-7371717862294852586 finished with status DONE.
Load test results for test (ID): 8cdbdbfb-6fcb-49fa-a861-a1bb58fd0ad5 and timestamp: 2022-05-10T12:32:50.673000000Z:
Metric: Value:
dataflow_v2_java17_runtime_sec 5813.232
dataflow_v2_java17_total_bytes_count 6.0274294E9
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220510123038
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:681c0d5f2a230de72485a61d99b98d34569d0303dda01320734c2ebe64c4a3da
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220510123038]
- referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:681c0d5f2a230de72485a61d99b98d34569d0303dda01320734c2ebe64c4a3da]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220510123038] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:681c0d5f2a230de72485a61d99b98d34569d0303dda01320734c2ebe64c4a3da])].
Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:e016287d22625030f983cbf6f23e489ff06d1e82e3b1276c012f318dcaa366bf
Digests:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:e016287d22625030f983cbf6f23e489ff06d1e82e3b1276c012f318dcaa366bf
ERROR: (gcloud.container.images.delete) Not found: response: {'docker-distribution-api-version': 'registry/2.0', 'content-type': 'application/json', 'date': 'Tue, 10 May 2022 14:14:25 GMT', 'server': 'Docker Registry', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'transfer-encoding': 'chunked', 'status': '404', 'content-length': '168', '-content-encoding': 'gzip'}
Failed to compute blob liveness for manifest: 'sha256:e016287d22625030f983cbf6f23e489ff06d1e82e3b1276c012f318dcaa366bf': None
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/ws/src/runners/google-cloud-dataflow-java/build.gradle'> line: 297
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:cleanUpDockerJavaImages'.
> Process 'command './scripts/cleanup_untagged_gcr_images.sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 1h 44m
110 actionable tasks: 73 executed, 33 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/iters5gdlxc5s
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17 #140
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/140/display/redirect?page=changes>
Changes:
[elias.segundo] Changing elegibility to AllNodeElegibility
[chamikaramj] Adds code reviewers for GCP I/O connectors and KafkaIO to Beam OWNERS
------------------------------------------
[...truncated 50.04 KB...]
e93827457889: Preparing
08fa02ce37eb: Preparing
a037458de4e0: Preparing
bafdbe68e4ae: Preparing
a13c519c6361: Preparing
4558b07d7c6c: Waiting
7572dcf244a7: Waiting
a037458de4e0: Waiting
bafdbe68e4ae: Waiting
a13c519c6361: Waiting
76cbfa6ab6de: Waiting
3bc383470c05: Waiting
212e3bed8e24: Waiting
a1cdc24dc423: Waiting
08fa02ce37eb: Waiting
e93827457889: Waiting
e71737e48960: Waiting
9eded4dea7e1: Waiting
d25a54d106dd: Waiting
65c0e27593f2: Waiting
b0a0cfc0c1c7: Pushed
99b1b0c017c9: Pushed
70d12e40179b: Pushed
74d7ad42a174: Pushed
a7b750c44030: Pushed
76cbfa6ab6de: Pushed
212e3bed8e24: Pushed
65c0e27593f2: Pushed
e71737e48960: Pushed
9eded4dea7e1: Pushed
a1cdc24dc423: Pushed
3bc383470c05: Layer already exists
e93827457889: Layer already exists
08fa02ce37eb: Layer already exists
4fdbf0a79d70: Pushed
a037458de4e0: Layer already exists
bafdbe68e4ae: Layer already exists
a13c519c6361: Layer already exists
4558b07d7c6c: Pushed
7572dcf244a7: Pushed
d25a54d106dd: Pushed
20220509123035: digest: sha256:ae661c08a40bbf8a47368c1570d2cbdb9dbb0865ef587bd35d60f43cbee2a87d size: 4729
> Task :sdks:java:testing:load-tests:run
May 09, 2022 12:32:39 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 09, 2022 12:32:39 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 222 files. Enable logging at DEBUG level to see which files will be staged.
May 09, 2022 12:32:40 PM org.apache.beam.sdk.Pipeline validate
WARNING: The following transforms do not have stable unique names: Window.Into()
May 09, 2022 12:32:40 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
May 09, 2022 12:32:43 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 222 files from PipelineOptions.filesToStage to staging location to prepare for execution.
May 09, 2022 12:32:44 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 222 files cached, 0 files newly uploaded in 0 seconds
May 09, 2022 12:32:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/
May 09, 2022 12:32:44 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <120277 bytes, hash 703fa7d7284d03a6724f18a00f996b599f49c2d0600f6d8db28805bcee16d47a> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-cD-n1yhNA6ZyTxigD5lrWZ9JwtBgD22NsogFvO4W1Ho.pb
May 09, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1
May 09, 2022 12:32:45 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@751ae8a4, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@235d659c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4232b34a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2da16263, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@f5ce0bb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@47e51549, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@101a461c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@360e9c06, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5ebffb44, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@311ff287, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7377781e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@31db34da, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@109f8c7e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5edacf20, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@16a5eb6d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7e307087, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1220ef43, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1a8b81e8, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@234cff57, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5e8507f1]
May 09, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/StripIds as step s2
May 09, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (input) as step s3
May 09, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s4
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s5
May 09, 2022 12:32:46 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bf54172, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2c9a6717, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7b3cde6f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6d091cad, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7c663eaf, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bb5ceb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1e692555, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3ba0ae41, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@76fe6cdc, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2ffb3aec, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@786ff1cb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@46039a21, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@431e86b1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@35c4e864, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@32a2a6be, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@682af059, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5f36c8e3, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4da39ca9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6a9344f5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5584d9c6]
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/StripIds as step s6
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (co-input) as step s7
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()2/Window.Assign as step s8
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable0 as step s9
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable1 as step s10
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/Flatten as step s11
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/GBK as step s12
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s13
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate as step s14
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect total bytes as step s15
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics as step s16
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.40.0-SNAPSHOT
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-09_05_32_46-14665142041617253802?project=apache-beam-testing
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-05-09_05_32_46-14665142041617253802
May 09, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2022-05-09_05_32_46-14665142041617253802
May 09, 2022 12:32:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-05-09T12:32:51.088Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java170dataflow0v20streaming0cogbk01-jenkins-05-ithr. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions
May 09, 2022 12:32:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:56.474Z: Worker configuration: e2-standard-2 in us-central1-b.
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.352Z: Expanding SplittableParDo operations into optimizable parts.
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.408Z: Expanding CollectionToSingleton operations into optimizable parts.
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.481Z: Expanding CoGroupByKey operations into optimizable parts.
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.573Z: Expanding SplittableProcessKeyed operations into optimizable parts.
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.613Z: Expanding GroupByKey operations into streaming Read/Write steps
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.689Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.803Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.838Z: Unzipping flatten CoGroupByKey-Flatten for input CoGroupByKey-MakeUnionTable0-ParMultiDo-ConstructUnionTable-.output
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.881Z: Fusing unzipped copy of CoGroupByKey/GBK/WriteStream, through flatten CoGroupByKey/Flatten, into producer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.914Z: Fusing consumer CoGroupByKey/GBK/WriteStream into CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.954Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:57.985Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.023Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.058Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.093Z: Fusing consumer Collect start time metrics (input)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.124Z: Fusing consumer Window.Into()/Window.Assign into Collect start time metrics (input)/ParMultiDo(TimeMonitor)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.158Z: Fusing consumer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) into Window.Into()/Window.Assign
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.194Z: Fusing consumer Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read co-input/Impulse
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.227Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.251Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.284Z: Fusing consumer Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.319Z: Fusing consumer Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) into Read co-input/ParDo(StripIds)/ParMultiDo(StripIds)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.353Z: Fusing consumer Window.Into()2/Window.Assign into Collect start time metrics (co-input)/ParMultiDo(TimeMonitor)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.400Z: Fusing consumer CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) into Window.Into()2/Window.Assign
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.429Z: Fusing consumer CoGroupByKey/GBK/MergeBuckets into CoGroupByKey/GBK/ReadStream
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.456Z: Fusing consumer CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) into CoGroupByKey/GBK/MergeBuckets
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.492Z: Fusing consumer Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) into CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.525Z: Fusing consumer Collect total bytes/ParMultiDo(ByteMonitor) into Ungroup and reiterate/ParMultiDo(UngroupAndReiterate)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.575Z: Fusing consumer Collect end time metrics/ParMultiDo(TimeMonitor) into Collect total bytes/ParMultiDo(ByteMonitor)
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.758Z: Running job using Streaming Engine
May 09, 2022 12:33:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:32:58.982Z: Starting 5 ****s in us-central1-b...
May 09, 2022 12:33:22 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:33:21.818Z: Autoscaling: Raised the number of ****s to 4 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 09, 2022 12:33:22 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:33:21.846Z: Resized **** pool to 4, though goal was 5. This could be a quota issue.
May 09, 2022 12:33:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:33:29.180Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
May 09, 2022 12:33:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:33:32.071Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 09, 2022 12:34:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T12:34:33.040Z: Workers have started successfully.
May 09, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T16:01:00.503Z: Cancel request is committed for workflow job: 2022-05-09_05_32_46-14665142041617253802.
May 09, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T16:01:00.620Z: Cleaning up.
May 09, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T16:01:00.718Z: Stopping **** pool...
May 09, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T16:01:00.768Z: Stopping **** pool...
May 09, 2022 4:01:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T16:01:32.713Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
May 09, 2022 4:01:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-09T16:01:32.750Z: Worker pool stopped.
May 09, 2022 4:01:41 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2022-05-09_05_32_46-14665142041617253802 finished with status CANCELLED.
Load test results for test (ID): a6fbb7f6-1e3f-46ae-9feb-1181e26d3207 and timestamp: 2022-05-09T12:32:40.038000000Z:
Metric: Value:
dataflow_v2_java17_runtime_sec 12338.085
dataflow_v2_java17_total_bytes_count 2.86964364E10
Exception in thread "main" java.lang.RuntimeException: Invalid job state: CANCELLED.
at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51)
at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.run(CoGroupByKeyLoadTest.java:62)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.main(CoGroupByKeyLoadTest.java:157)
> Task :sdks:java:testing:load-tests:run FAILED
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220509123035
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ae661c08a40bbf8a47368c1570d2cbdb9dbb0865ef587bd35d60f43cbee2a87d
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220509123035]
- referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ae661c08a40bbf8a47368c1570d2cbdb9dbb0865ef587bd35d60f43cbee2a87d]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220509123035] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ae661c08a40bbf8a47368c1570d2cbdb9dbb0865ef587bd35d60f43cbee2a87d])].
Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ae661c08a40bbf8a47368c1570d2cbdb9dbb0865ef587bd35d60f43cbee2a87d
Digests:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ae661c08a40bbf8a47368c1570d2cbdb9dbb0865ef587bd35d60f43cbee2a87d
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:ae661c08a40bbf8a47368c1570d2cbdb9dbb0865ef587bd35d60f43cbee2a87d].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:load-tests:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 3h 31m 27s
110 actionable tasks: 73 executed, 33 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/5fdkmmdldigby
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17 #139
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/139/display/redirect>
Changes:
------------------------------------------
[...truncated 50.84 KB...]
08fa02ce37eb: Preparing
a037458de4e0: Preparing
bafdbe68e4ae: Preparing
a13c519c6361: Preparing
fced9607bc47: Waiting
e91b08f16562: Waiting
713082863eee: Waiting
08e8a29c256a: Waiting
00d4a583dbe4: Waiting
875a60575306: Waiting
c92dc92e5b42: Waiting
721b8ef4b4ba: Waiting
b900d4df52f8: Waiting
5b26c8721c69: Waiting
3bc383470c05: Waiting
bafdbe68e4ae: Waiting
a13c519c6361: Waiting
e93827457889: Waiting
08fa02ce37eb: Waiting
a037458de4e0: Waiting
fbe1b45afd3e: Pushed
c8df78b28fbb: Pushed
e023246e7a9a: Pushed
d580cb531531: Pushed
de4ea1bb1e74: Pushed
875a60575306: Pushed
721b8ef4b4ba: Pushed
b900d4df52f8: Pushed
5b26c8721c69: Pushed
713082863eee: Pushed
e91b08f16562: Pushed
fced9607bc47: Pushed
e93827457889: Layer already exists
08fa02ce37eb: Layer already exists
3bc383470c05: Layer already exists
a037458de4e0: Layer already exists
bafdbe68e4ae: Layer already exists
a13c519c6361: Layer already exists
00d4a583dbe4: Pushed
c92dc92e5b42: Pushed
08e8a29c256a: Pushed
20220508123043: digest: sha256:3e2bc6f9b92f30686d7581e866cc3457a9e65cb36f5ee4f62689dd7b0f578fb4 size: 4729
> Task :sdks:java:testing:load-tests:run
May 08, 2022 12:33:00 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 08, 2022 12:33:01 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 222 files. Enable logging at DEBUG level to see which files will be staged.
May 08, 2022 12:33:01 PM org.apache.beam.sdk.Pipeline validate
WARNING: The following transforms do not have stable unique names: Window.Into()
May 08, 2022 12:33:01 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
May 08, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 222 files from PipelineOptions.filesToStage to staging location to prepare for execution.
May 08, 2022 12:33:06 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 222 files cached, 0 files newly uploaded in 0 seconds
May 08, 2022 12:33:06 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/
May 08, 2022 12:33:06 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <120277 bytes, hash 6c4231b8b37219127e2cbe678df05f694b012a4e308249a674ec8a16db5c1949> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-bEIxuLNyGRJ-LL5njfBfaUsBKk4wgkmmdOyKFttcGUk.pb
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1
May 08, 2022 12:33:08 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@751ae8a4, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@235d659c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4232b34a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2da16263, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@f5ce0bb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@47e51549, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@101a461c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@360e9c06, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5ebffb44, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@311ff287, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7377781e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@31db34da, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@109f8c7e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5edacf20, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@16a5eb6d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7e307087, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1220ef43, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1a8b81e8, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@234cff57, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5e8507f1]
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/StripIds as step s2
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (input) as step s3
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s4
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s5
May 08, 2022 12:33:08 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bf54172, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2c9a6717, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7b3cde6f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6d091cad, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7c663eaf, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bb5ceb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1e692555, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3ba0ae41, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@76fe6cdc, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2ffb3aec, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@786ff1cb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@46039a21, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@431e86b1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@35c4e864, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@32a2a6be, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@682af059, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5f36c8e3, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4da39ca9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6a9344f5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5584d9c6]
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/StripIds as step s6
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (co-input) as step s7
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()2/Window.Assign as step s8
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable0 as step s9
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable1 as step s10
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/Flatten as step s11
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/GBK as step s12
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s13
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate as step s14
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect total bytes as step s15
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics as step s16
May 08, 2022 12:33:08 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.40.0-SNAPSHOT
May 08, 2022 12:33:09 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-08_05_33_08-12380385818833013014?project=apache-beam-testing
May 08, 2022 12:33:09 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-05-08_05_33_08-12380385818833013014
May 08, 2022 12:33:09 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2022-05-08_05_33_08-12380385818833013014
May 08, 2022 12:33:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-05-08T12:33:14.423Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java170dataflow0v20streaming0cogbk01-jenkins-05-v0tj. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:25.065Z: Worker configuration: e2-standard-2 in us-central1-b.
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:25.642Z: Expanding SplittableParDo operations into optimizable parts.
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:25.737Z: Expanding CollectionToSingleton operations into optimizable parts.
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:25.818Z: Expanding CoGroupByKey operations into optimizable parts.
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:25.888Z: Expanding SplittableProcessKeyed operations into optimizable parts.
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:25.917Z: Expanding GroupByKey operations into streaming Read/Write steps
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:25.983Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.099Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.130Z: Unzipping flatten CoGroupByKey-Flatten for input CoGroupByKey-MakeUnionTable0-ParMultiDo-ConstructUnionTable-.output
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.164Z: Fusing unzipped copy of CoGroupByKey/GBK/WriteStream, through flatten CoGroupByKey/Flatten, into producer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.199Z: Fusing consumer CoGroupByKey/GBK/WriteStream into CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.230Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.270Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.306Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.340Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.371Z: Fusing consumer Collect start time metrics (input)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.410Z: Fusing consumer Window.Into()/Window.Assign into Collect start time metrics (input)/ParMultiDo(TimeMonitor)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.443Z: Fusing consumer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) into Window.Into()/Window.Assign
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.476Z: Fusing consumer Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read co-input/Impulse
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.509Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.542Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.568Z: Fusing consumer Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.605Z: Fusing consumer Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) into Read co-input/ParDo(StripIds)/ParMultiDo(StripIds)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.639Z: Fusing consumer Window.Into()2/Window.Assign into Collect start time metrics (co-input)/ParMultiDo(TimeMonitor)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.736Z: Fusing consumer CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) into Window.Into()2/Window.Assign
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.771Z: Fusing consumer CoGroupByKey/GBK/MergeBuckets into CoGroupByKey/GBK/ReadStream
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.802Z: Fusing consumer CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) into CoGroupByKey/GBK/MergeBuckets
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.836Z: Fusing consumer Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) into CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.872Z: Fusing consumer Collect total bytes/ParMultiDo(ByteMonitor) into Ungroup and reiterate/ParMultiDo(UngroupAndReiterate)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:26.897Z: Fusing consumer Collect end time metrics/ParMultiDo(TimeMonitor) into Collect total bytes/ParMultiDo(ByteMonitor)
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:27.139Z: Running job using Streaming Engine
May 08, 2022 12:33:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:27.534Z: Starting 5 ****s in us-central1-b...
May 08, 2022 12:33:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:41.960Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
May 08, 2022 12:33:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:49.913Z: Autoscaling: Raised the number of ****s to 4 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 08, 2022 12:33:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:33:49.939Z: Resized **** pool to 4, though goal was 5. This could be a quota issue.
May 08, 2022 12:34:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:34:00.147Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 08, 2022 12:34:58 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T12:34:57.660Z: Workers have started successfully.
May 08, 2022 4:01:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T16:01:12.049Z: Cancel request is committed for workflow job: 2022-05-08_05_33_08-12380385818833013014.
May 08, 2022 4:01:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T16:01:12.227Z: Cleaning up.
May 08, 2022 4:01:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T16:01:12.337Z: Stopping **** pool...
May 08, 2022 4:01:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T16:01:12.380Z: Stopping **** pool...
May 08, 2022 4:01:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T16:01:45.060Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
May 08, 2022 4:01:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-08T16:01:45.128Z: Worker pool stopped.
May 08, 2022 4:01:52 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2022-05-08_05_33_08-12380385818833013014 finished with status CANCELLED.
Load test results for test (ID): 6898ae29-0c8e-440a-aed1-3e8098cdf3b0 and timestamp: 2022-05-08T12:33:01.540000000Z:
Metric: Value:
dataflow_v2_java17_runtime_sec 12302.848
dataflow_v2_java17_total_bytes_count 2.44242771E10
Exception in thread "main" java.lang.RuntimeException: Invalid job state: CANCELLED.
at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51)
at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.run(CoGroupByKeyLoadTest.java:62)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.main(CoGroupByKeyLoadTest.java:157)
> Task :sdks:java:testing:load-tests:run FAILED
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220508123043
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:3e2bc6f9b92f30686d7581e866cc3457a9e65cb36f5ee4f62689dd7b0f578fb4
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220508123043]
- referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:3e2bc6f9b92f30686d7581e866cc3457a9e65cb36f5ee4f62689dd7b0f578fb4]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220508123043] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:3e2bc6f9b92f30686d7581e866cc3457a9e65cb36f5ee4f62689dd7b0f578fb4])].
Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:3e2bc6f9b92f30686d7581e866cc3457a9e65cb36f5ee4f62689dd7b0f578fb4
Digests:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:3e2bc6f9b92f30686d7581e866cc3457a9e65cb36f5ee4f62689dd7b0f578fb4
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:3e2bc6f9b92f30686d7581e866cc3457a9e65cb36f5ee4f62689dd7b0f578fb4].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:load-tests:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 3h 31m 31s
110 actionable tasks: 73 executed, 33 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/a3nj7pq4szqmc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17 #138
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/138/display/redirect?page=changes>
Changes:
[kevinsijo] Setting up a basic directory
[kevinsijo] Mirroring Python SDK's directory structure
[kerrydc] Adds initial tests
[kevinsijo] 'runners' is the correct directory name
[Pablo Estrada] sketching the core API for JS SDK
[jonathanlui] add .gitignore for node/ts project
[Robert Bradshaw] Worker directory.
[Robert Bradshaw] Fix complile errors with explicit any for callables.
[Robert Bradshaw] Add worker entry point.
[Robert Bradshaw] Add proto generation code.
[Robert Bradshaw] Add generated proto files.
[Robert Bradshaw] Attempts to get ts protos to compile.
[Robert Bradshaw] Exclude ts protos for now.
[Robert Bradshaw] More changes to get ts protos working.
[Robert Bradshaw] Update scripts and config to get protos compiling.
[Robert Bradshaw] Update geenrated files.
[jonathanlui] add build and clean script to compile ts
[Robert Bradshaw] Generate server for loopback worker.
[Robert Bradshaw] Generated grpc servers for loopback.
[Robert Bradshaw] Add typescript formatter.
[Robert Bradshaw] Loopback server (that does nothing).
[Robert Bradshaw] Working server.
[Pablo Estrada] Starting expansion of primitive transforms
[Pablo Estrada] Starting to implement and support standard coders
[Robert Bradshaw] Also generate grpc clients.
[Robert Bradshaw] Basic implementation of worker harness.
[Pablo Estrada] fix the build
[Robert Bradshaw] Add some missing files for worker harness.
[Robert Bradshaw] Refactor operators to use registration.
[jonathanlui] enable ts in mocha
[jonathanlui] update readme
[jonathanlui] --save-dev @types/mocha
[jonathanlui] translate core_test.js to typescript
[Robert Bradshaw] Encapsulate worker service in a class.
[Kenneth Knowles] Port standard_coders_test to typescript (superficially)
[Pablo Estrada] Starting the proto translation of Impulse, ParDo, GBK
[Robert Bradshaw] Add some tests for the worker code.
[Robert Bradshaw] Fixing old lock file error.
[Pablo Estrada] Adding transform names and fixing GBK coder issue
[Robert Bradshaw] npx tsfmt -r src/apache_beam/base.ts src/apache_beam/transforms/core.ts
[Kenneth Knowles] switch to import style require() statements
[Kenneth Knowles] Add Coder interface using protobufjs classes
[Kenneth Knowles] BytesCoder with some failures
[noreply] Added GeneralObjectCoder and using it as coder for most transforms (#9)
[Kenneth Knowles] Fix order of arguments to deepEqual
[Kenneth Knowles] Encode expected encoding as binary
[Robert Bradshaw] Refactor API to allow for composites.
[jrmccluskey] Initial setup for automated Java expansion startup
[jrmccluskey] Update exp_service.ts
[Kenneth Knowles] Fix up coder deserialization
[Robert Bradshaw] Simplify GBK coder computation.
[Robert Bradshaw] Remove top-level PValue.
[Pablo Estrada] Make tests green
[Robert Bradshaw] Rename PValueish to PValue.
[jonathanlui] node runner
[jonathanlui] whitespaces
[Robert Bradshaw] Make Runner.run async.
[jonathanlui] bson and fast-deep-equal should not be listed as devdependency
[jrmccluskey] Add basic Dockerfile that starts ExternalWorkerPool
[Robert Bradshaw] Direct runner.
[kevinsijo] Testing expansion service communication
[Robert Bradshaw] Added flatten, assertion checkers.
[Pablo Estrada] progress on basic coders
[Robert Bradshaw] Fixing the build.
[Robert Bradshaw] Cleanup, simplify access.
[Pablo Estrada] Adding limited support for KVCoder and IterableCoder
[Robert Bradshaw] Introduce PipelineContext.
[Robert Bradshaw] Add toProto to all coders.
[Robert Bradshaw] Some work with coders.
[Robert Bradshaw] Remove debug logging.
[Robert Bradshaw] Use coders over data channel.
[Kenneth Knowles] explicitly sequence sub-coder serializations
[Kenneth Knowles] no more need to extend FakeCoder
[Kenneth Knowles] actually advance reader
[Kenneth Knowles] autoformat
[Kenneth Knowles] protobufjs already can write and read signed varints
[Kenneth Knowles] with improved test harness, kv has many more failures
[Kenneth Knowles] read bytescoder from correct position
[Kenneth Knowles] no more fake coders
[Kenneth Knowles] varint examples all work
[Kenneth Knowles] simplify coder value parsing
[Kenneth Knowles] global window coder
[Kenneth Knowles] fix swapEndian32
[Robert Bradshaw] Add P(...) operator.
[kevinsijo] Implementing RowCoder encoding.
[jrmccluskey] remove unused container dir
[kevinsijo] Corrected sorting of encoded positions to reflect an argsort instead.
[Robert Bradshaw] Populate environments.
[kevinsijo] Implementing RowCoder decoding.
[Kenneth Knowles] preliminary unbounded iterable coder
[Kenneth Knowles] friendlier description of standard coder test case
[Kenneth Knowles] fix test harness; iterable works
[jrmccluskey] first pass at boot.go
[jonathanlui] update package-lock.json
[jonathanlui] make NodeRunner a subclass of Runner
[jonathanlui] add waitUntilFinish interface member
[Pablo Estrada] Adding double coder
[Kenneth Knowles] scaffolding for windowed values
[Pablo Estrada] Adding type information to PColleciton and PTransform
[jonathanlui] fix direct runner
[Pablo Estrada] Adding typing information for DoFns
[Kenneth Knowles] add interval window
[Robert Bradshaw] Export PValue.
[Robert Bradshaw] Add CombineFn interface.
[Robert Bradshaw] Typed flatten.
[jonathanlui] add runAsync method to base.Runner
[Kenneth Knowles] add Long package
[Pablo Estrada] Adding more types. Making PValue typed
[Kenneth Knowles] instant coder draft
[Robert Bradshaw] Return job state from direct runner.
[Kenneth Knowles] type instant = long
[jonathanlui] implement NodeRunner.runPipeline
[Kenneth Knowles] autoformat
[kevinsijo] Completed implementation of basic row coder
[Kenneth Knowles] Fix IntervalWindowCoder, almost
[Kenneth Knowles] fix interval window coder
[Kenneth Knowles] autoformat
[Robert Bradshaw] loopback runner works
[Kenneth Knowles] move core element types into values.ts
[Kenneth Knowles] just build object directly to be cool
[Robert Bradshaw] GBK working on ULR.
[Robert Bradshaw] Async transforms.
[Robert Bradshaw] External transform grpah splicing.
[Kenneth Knowles] progress on windowed value: paneinfo encoding
[Robert Bradshaw] Fix merge.
[Robert Bradshaw] autoformat
[Kenneth Knowles] full windowed value coder
[kerrydc] Updates tests to use correct types, adds generics where needed to DoFns
[Robert Bradshaw] Add serialization librarires.'
[Robert Bradshaw] Add Split() PTransform, for producing multiple outputs from a single
[Robert Bradshaw] Schema-encoded external payloads.
[kevinsijo] Adding Schema inference from JSON
[Pablo Estrada] Removing unused directories
[Pablo Estrada] Support for finishBundle and improving typing annotations.
[Pablo Estrada] A base implementation of combiners with GBK/ParDo
[Robert Bradshaw] Fully propagate windowing information in both remote and direct runner.
[Robert Bradshaw] Make args and kwargs optional for python external transform.
[Robert Bradshaw] Infer schema for external transforms.
[Pablo Estrada] Implementing a custom combine fn as an example. Small fixes
[Robert Bradshaw] Fix missing windowing information in combiners.
[Robert Bradshaw] PostShuffle needn't group by key as that's already done.
[Robert Bradshaw] Guard pre-combine for global window only.
[Robert Bradshaw] WindowInto
[Robert Bradshaw] Fix optional kwargs.
[Robert Bradshaw] A couple of tweaks for js + py
[Robert Bradshaw] Add windowing file.
[Robert Bradshaw] CombineBy transform, stand-alone WordCount.
[Robert Bradshaw] cleanup
[Robert Bradshaw] Actually fix optional external kwargs.
[Robert Bradshaw] Demo2, textio read.
[Robert Bradshaw] Add command lines for starting up the servers.
[Robert Bradshaw] Run prettier on the full codebase.
[Robert Bradshaw] Update deps.
[Pablo Estrada] Adding docstrings for core.ts. Prettier dependency
[Pablo Estrada] Documenting coder interfaces
[Pablo Estrada] Added documentation for a few standard coders
[Robert Bradshaw] Unified grouping and combining.
[Robert Bradshaw] Allow PCollection ids to be lazy.
[Robert Bradshaw] Reorganize module structure.
[Robert Bradshaw] A couple more renames.
[Robert Bradshaw] Simplify.
[Robert Bradshaw] Consolidation.
[Robert Bradshaw] Fix build.
[Robert Bradshaw] Add optional context to ParDo.
[Robert Bradshaw] fixup: iterable coder endian sign issue
[Robert Bradshaw] omit context for map(console.log)
[Robert Bradshaw] Fix ReadFromText coders.
[Robert Bradshaw] Flesh out README with overview and current state.
[noreply] Readme typo
[Robert Bradshaw] Two more TODOs.
[noreply] Add a pointer to the example wordcount to the readme.
[Pablo Estrada] Documenting coders and implementing unknown-length method
[Robert Bradshaw] UIID dependency.
[Robert Bradshaw] Artifact handling.
[Robert Bradshaw] Properly wait on data channel for bundle completion.
[Robert Bradshaw] Automatic java expansion service startup.
[Robert Bradshaw] Process promises.
[Robert Bradshaw] Implement side inputs.
[Robert Bradshaw] Cleanup.
[Robert Bradshaw] Put complex constext stuff in its own file.
[Robert Bradshaw] Rename BoundedWindow to just Window.
[Robert Bradshaw] Alternative splitter class.
[Pablo Estrada] Documenting internal functions
[Robert Bradshaw] Take a pass clarifying the TODOs.
[Robert Bradshaw] Sql transform wrapper.
[Robert Bradshaw] Incorporate some feedback into the TODOs.
[Robert Bradshaw] More TODOs.
[Robert Bradshaw] Remove app placeholder.
[Robert Bradshaw] Apache license headers.
[Robert Bradshaw] More TODOs
[jankuehle] Suggestions for TypeScript todos
[dannymccormick] Add actions for typescript sdk
[dannymccormick] Fix test command
[noreply] Add missing version
[dannymccormick] Fix codecovTest command
[noreply] Only do prettier check on linux
[noreply] Only get codecov on linux
[Robert Bradshaw] Resolve some comments.
[Robert Bradshaw] Fix compile errors.
[Robert Bradshaw] Prettier.
[Robert Bradshaw] Re-order expandInternal arguments pending unification.
[Robert Bradshaw] More consistent and stricter PTransform naming.
[Robert Bradshaw] Notes on explicit, if less idiomatic, use of classes.
[Robert Bradshaw] Let DoFn be an interface rather than a class.
[Robert Bradshaw] Provide DoFn context to start and finish bundle.
[Robert Bradshaw] Optional promise code simplification.
[Robert Bradshaw] Cleanup todos.
[Robert Bradshaw] Avoid any type where not needed.
[Robert Bradshaw] Apache RAT excludes for typescript.
[Robert Bradshaw] Remove empty READMEs.
[Robert Bradshaw] Add licences statement to readme files.
[Robert Bradshaw] More RAT fixes.
[Robert Bradshaw] Another unsupported coder.
[Robert Bradshaw] Remove debugging code.
[noreply] Fix automatic naming with code coverage.
[Robert Bradshaw] Coders cleanup.
[Robert Bradshaw] Add tests for RowCoder.
[Robert Bradshaw] Normalize capitalization, comments.
[Robert Bradshaw] Install typescript closure packages.
[Robert Bradshaw] npm audit fix
[Robert Bradshaw] Move more imports out of base.
[Robert Bradshaw] Changes needed to compile with ts closure plugin.
[Robert Bradshaw] Use ttsc and ts-closure-transform plugin.
[Robert Bradshaw] Serialization registration to actually get serialization working.
[Robert Bradshaw] Container images working on local runner.
[Robert Bradshaw] Add a portable job server that proxies the Dataflow backend. (#17189)
[Robert Bradshaw] Improvements to dataflow job service for non-Python jobs.
[Robert Bradshaw] Get dataflow working.
[Robert Bradshaw] User friendly pipeline options.
[Robert Bradshaw] Less classes, more functions.
[Robert Bradshaw] Add new nullable standard coder.
[Robert Bradshaw] Make Apache Rat happy.
[Robert Bradshaw] Disable broken codecov.
[Robert Bradshaw] Remove last uses of base.ts.
[Robert Bradshaw] Remove unneedd file.
[Robert Bradshaw] Remove more uneeded/unused files.
[Robert Bradshaw] Cleanup tests.
[Robert Bradshaw] Minor cleanups to coder tests.
[noreply] Quote pip install package name
[noreply] [BEAM-14374] Fix module import error in FullyQualifiedNamedTransform
[Robert Bradshaw] Addressing issues from the review.
[noreply] Apply suggestions from code review.
[Robert Bradshaw] Post-merge fixes.
[dannymccormick] Delete tags.go
[Robert Bradshaw] Update tests to use our actual serialization libraries.
[Robert Bradshaw] Another pass at TODOs, removing finished items.
[Heejong Lee] [BEAM-14146] Python Streaming job failing to drain with BigQueryIO write
[Kenneth Knowles] Add parameter for service account impersonation in GCP credentials
[Heejong Lee] add test
[noreply] Merge pull request #17490 from [BEAM-14370] [Website] Add new page about
[noreply] [BEAM-14332] Refactored cluster management for Flink on Dataproc
[noreply] [BEAM-13988] Update mtime to use time.UnixMilli() calls (#17578)
[noreply] Fixing patching error on missing dependencies (#17564)
[noreply] Merge pull request #17517 from [BEAM-14383] Improve "FailedRows" errors
[Heejong Lee] add test without mock
------------------------------------------
[...truncated 50.18 KB...]
2d05db1e2a4b: Preparing
3bc383470c05: Preparing
e93827457889: Preparing
08fa02ce37eb: Preparing
a037458de4e0: Preparing
662540b7db15: Waiting
1b1b55724774: Waiting
bafdbe68e4ae: Preparing
0e2e00df1023: Waiting
d507830c9b22: Waiting
a13c519c6361: Preparing
08fa02ce37eb: Waiting
82227587df8f: Waiting
e93827457889: Waiting
3bc383470c05: Waiting
bafdbe68e4ae: Waiting
a13c519c6361: Waiting
1252636fdc5b: Waiting
b0bdfb1e5c67: Waiting
8db9d37c9268: Waiting
2d05db1e2a4b: Waiting
a037458de4e0: Waiting
413656da2079: Pushed
98faf80f3452: Pushed
94899fdeb521: Pushed
3c9402bf3f9b: Pushed
1e44127ca56a: Pushed
662540b7db15: Pushed
104aa3e0eab7: Pushed
82227587df8f: Pushed
d507830c9b22: Pushed
1252636fdc5b: Pushed
1b1b55724774: Pushed
3bc383470c05: Layer already exists
e93827457889: Layer already exists
b0bdfb1e5c67: Pushed
08fa02ce37eb: Layer already exists
a037458de4e0: Layer already exists
bafdbe68e4ae: Layer already exists
a13c519c6361: Layer already exists
8db9d37c9268: Pushed
2d05db1e2a4b: Pushed
0e2e00df1023: Pushed
20220507123039: digest: sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a size: 4729
> Task :sdks:java:testing:load-tests:run
May 07, 2022 12:32:44 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 07, 2022 12:32:44 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 222 files. Enable logging at DEBUG level to see which files will be staged.
May 07, 2022 12:32:45 PM org.apache.beam.sdk.Pipeline validate
WARNING: The following transforms do not have stable unique names: Window.Into()
May 07, 2022 12:32:45 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
May 07, 2022 12:32:48 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 222 files from PipelineOptions.filesToStage to staging location to prepare for execution.
May 07, 2022 12:32:48 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 222 files cached, 0 files newly uploaded in 0 seconds
May 07, 2022 12:32:48 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/
May 07, 2022 12:32:48 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <120277 bytes, hash a6ab5cbe90343949e56d702d9a136db2fa719819f168bdcc66299093c41072f1> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-pqtcvpA0OUnlbXAtmhNtsvpxmBnxaL3MZimQk8QQcvE.pb
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1
May 07, 2022 12:32:50 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@751ae8a4, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@235d659c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4232b34a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2da16263, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@f5ce0bb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@47e51549, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@101a461c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@360e9c06, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5ebffb44, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@311ff287, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7377781e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@31db34da, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@109f8c7e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5edacf20, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@16a5eb6d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7e307087, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1220ef43, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1a8b81e8, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@234cff57, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5e8507f1]
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/StripIds as step s2
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (input) as step s3
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s4
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s5
May 07, 2022 12:32:50 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bf54172, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2c9a6717, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7b3cde6f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6d091cad, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7c663eaf, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bb5ceb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1e692555, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3ba0ae41, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@76fe6cdc, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2ffb3aec, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@786ff1cb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@46039a21, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@431e86b1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@35c4e864, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@32a2a6be, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@682af059, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5f36c8e3, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4da39ca9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6a9344f5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5584d9c6]
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/StripIds as step s6
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (co-input) as step s7
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()2/Window.Assign as step s8
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable0 as step s9
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable1 as step s10
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/Flatten as step s11
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/GBK as step s12
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s13
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate as step s14
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect total bytes as step s15
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics as step s16
May 07, 2022 12:32:50 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.40.0-SNAPSHOT
May 07, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-07_05_32_50-18293218377134002814?project=apache-beam-testing
May 07, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-05-07_05_32_50-18293218377134002814
May 07, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2022-05-07_05_32_50-18293218377134002814
May 07, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-05-07T12:32:54.377Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java170dataflow0v20streaming0cogbk01-jenkins-05-e677. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions
May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:03.852Z: Worker configuration: e2-standard-2 in us-central1-b.
May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:04.691Z: Expanding SplittableParDo operations into optimizable parts.
May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:04.716Z: Expanding CollectionToSingleton operations into optimizable parts.
May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:04.786Z: Expanding CoGroupByKey operations into optimizable parts.
May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:04.935Z: Expanding SplittableProcessKeyed operations into optimizable parts.
May 07, 2022 12:33:05 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.134Z: Expanding GroupByKey operations into streaming Read/Write steps
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.412Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.501Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.614Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.650Z: Unzipping flatten CoGroupByKey-Flatten for input CoGroupByKey-MakeUnionTable0-ParMultiDo-ConstructUnionTable-.output
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.687Z: Fusing unzipped copy of CoGroupByKey/GBK/WriteStream, through flatten CoGroupByKey/Flatten, into producer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.719Z: Fusing consumer CoGroupByKey/GBK/WriteStream into CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.753Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.786Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.820Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.851Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.884Z: Fusing consumer Collect start time metrics (input)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.917Z: Fusing consumer Window.Into()/Window.Assign into Collect start time metrics (input)/ParMultiDo(TimeMonitor)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.942Z: Fusing consumer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) into Window.Into()/Window.Assign
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:05.975Z: Fusing consumer Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read co-input/Impulse
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.008Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.041Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.101Z: Fusing consumer Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.123Z: Fusing consumer Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) into Read co-input/ParDo(StripIds)/ParMultiDo(StripIds)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.166Z: Fusing consumer Window.Into()2/Window.Assign into Collect start time metrics (co-input)/ParMultiDo(TimeMonitor)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.199Z: Fusing consumer CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) into Window.Into()2/Window.Assign
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.232Z: Fusing consumer CoGroupByKey/GBK/MergeBuckets into CoGroupByKey/GBK/ReadStream
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.266Z: Fusing consumer CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) into CoGroupByKey/GBK/MergeBuckets
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.300Z: Fusing consumer Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) into CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.333Z: Fusing consumer Collect total bytes/ParMultiDo(ByteMonitor) into Ungroup and reiterate/ParMultiDo(UngroupAndReiterate)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.365Z: Fusing consumer Collect end time metrics/ParMultiDo(TimeMonitor) into Collect total bytes/ParMultiDo(ByteMonitor)
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.562Z: Running job using Streaming Engine
May 07, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:06.802Z: Starting 5 ****s in us-central1-b...
May 07, 2022 12:33:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:33:32.472Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 07, 2022 12:34:41 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T12:34:39.607Z: Workers have started successfully.
May 07, 2022 1:13:26 PM org.apache.beam.sdk.metrics.MetricsEnvironment getCurrentContainer
WARNING: Reporting metrics are not supported in the current execution environment.
May 07, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:00:58.218Z: Cancel request is committed for workflow job: 2022-05-07_05_32_50-18293218377134002814.
May 07, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:00:59.490Z: Cleaning up.
May 07, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:00:59.579Z: Stopping **** pool...
May 07, 2022 4:01:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:00:59.623Z: Stopping **** pool...
May 07, 2022 4:01:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:01:35.077Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
May 07, 2022 4:01:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-07T16:01:35.117Z: Worker pool stopped.
May 07, 2022 4:01:45 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2022-05-07_05_32_50-18293218377134002814 finished with status CANCELLED.
Load test results for test (ID): d380543f-25c8-4015-8b36-36a9fa03eb93 and timestamp: 2022-05-07T12:32:45.055000000Z:
Metric: Value:
dataflow_v2_java17_runtime_sec 12349.746
dataflow_v2_java17_total_bytes_count 3.37999092E10
Exception in thread "main" java.lang.RuntimeException: Invalid job state: CANCELLED.
at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51)
at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.run(CoGroupByKeyLoadTest.java:62)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.main(CoGroupByKeyLoadTest.java:157)
> Task :sdks:java:testing:load-tests:run FAILED
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220507123039
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220507123039]
- referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220507123039] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a])].
Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a
Digests:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:a201febb4501343aed27f34781031d37aebfe84ff97646ce1d23d15df7f7d15a].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:load-tests:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 3h 31m 26s
110 actionable tasks: 73 executed, 33 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/icdmjakku6vew
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17 #137
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/137/display/redirect?page=changes>
Changes:
[zyichi] Move master readme.md to 2.40.0
[noreply] [BEAM-14173] Fix Go Loadtests on Dataflow & partial fix for Flink
[noreply] Upgrade python sdk container requirements. (#17549)
[noreply] Merge pull request #17497: [BEAM-11205] Update GCP Libraries BOM version
[noreply] [BEAM-12603] Add retry on grpc data channel and remove retry from test.
[noreply] Merge pull request #17359: [BEAM-14303] Add a way to exclude output
[noreply] [BEAM-14347] Allow users to optimize DoFn execution with a single
[noreply] [BEAM-5878] Add (failing) kwonly-argument test (#17509)
------------------------------------------
[...truncated 47.03 KB...]
6736907aa128: Pushed
3bc383470c05: Layer already exists
e93827457889: Layer already exists
e0251c2cfd97: Pushed
08fa02ce37eb: Layer already exists
a037458de4e0: Layer already exists
bafdbe68e4ae: Layer already exists
a13c519c6361: Layer already exists
c21785f068bb: Pushed
feb9704976ce: Pushed
a73b38700550: Pushed
20220506123132: digest: sha256:df7a799483a8dfcf0e4964b95e5dc6ef838d5908578aa974f5608b4b2f79498f size: 4729
> Task :sdks:java:testing:load-tests:run
May 06, 2022 12:35:55 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 06, 2022 12:35:56 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 222 files. Enable logging at DEBUG level to see which files will be staged.
May 06, 2022 12:35:57 PM org.apache.beam.sdk.Pipeline validate
WARNING: The following transforms do not have stable unique names: Window.Into()
May 06, 2022 12:35:57 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
May 06, 2022 12:36:03 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 222 files from PipelineOptions.filesToStage to staging location to prepare for execution.
May 06, 2022 12:36:05 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 222 files cached, 0 files newly uploaded in 2 seconds
May 06, 2022 12:36:05 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/
May 06, 2022 12:36:05 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <120277 bytes, hash 4c05446f1cc0b7cb866bcf3be6df3ca8b2912a2d7b39757dd38920d959660589> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-TAVEbxzAt8uGa8875t88qLKRKi17OXV904kg2VlmBYk.pb
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1
May 06, 2022 12:36:08 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@751ae8a4, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@235d659c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4232b34a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2da16263, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@f5ce0bb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@47e51549, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@101a461c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@360e9c06, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5ebffb44, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@311ff287, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7377781e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@31db34da, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@109f8c7e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5edacf20, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@16a5eb6d, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7e307087, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1220ef43, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1a8b81e8, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@234cff57, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5e8507f1]
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/StripIds as step s2
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (input) as step s3
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s4
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s5
May 06, 2022 12:36:08 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3b95a6db, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bf54172, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2c9a6717, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7b3cde6f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6d091cad, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7c663eaf, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bb5ceb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1e692555, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3ba0ae41, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@76fe6cdc, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2ffb3aec, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@786ff1cb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@46039a21, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@431e86b1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@35c4e864, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@32a2a6be, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@682af059, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5f36c8e3, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4da39ca9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6a9344f5]
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/StripIds as step s6
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (co-input) as step s7
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()2/Window.Assign as step s8
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable0 as step s9
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable1 as step s10
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/Flatten as step s11
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/GBK as step s12
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s13
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate as step s14
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect total bytes as step s15
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics as step s16
May 06, 2022 12:36:08 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.40.0-SNAPSHOT
May 06, 2022 12:36:09 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-06_05_36_09-16672906884944557169?project=apache-beam-testing
May 06, 2022 12:36:09 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-05-06_05_36_09-16672906884944557169
May 06, 2022 12:36:09 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2022-05-06_05_36_09-16672906884944557169
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-05-06T12:36:16.319Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java170dataflow0v20streaming0cogbk01-jenkins-05-eke9. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:22.945Z: Worker configuration: e2-standard-2 in us-central1-b.
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:23.746Z: Expanding SplittableParDo operations into optimizable parts.
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:23.775Z: Expanding CollectionToSingleton operations into optimizable parts.
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:23.839Z: Expanding CoGroupByKey operations into optimizable parts.
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:23.900Z: Expanding SplittableProcessKeyed operations into optimizable parts.
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:23.919Z: Expanding GroupByKey operations into streaming Read/Write steps
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:23.983Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.087Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.120Z: Unzipping flatten CoGroupByKey-Flatten for input CoGroupByKey-MakeUnionTable0-ParMultiDo-ConstructUnionTable-.output
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.154Z: Fusing unzipped copy of CoGroupByKey/GBK/WriteStream, through flatten CoGroupByKey/Flatten, into producer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.188Z: Fusing consumer CoGroupByKey/GBK/WriteStream into CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.220Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.254Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.288Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.332Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.352Z: Fusing consumer Collect start time metrics (input)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.379Z: Fusing consumer Window.Into()/Window.Assign into Collect start time metrics (input)/ParMultiDo(TimeMonitor)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.412Z: Fusing consumer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) into Window.Into()/Window.Assign
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.448Z: Fusing consumer Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read co-input/Impulse
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.470Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.502Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.528Z: Fusing consumer Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.559Z: Fusing consumer Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) into Read co-input/ParDo(StripIds)/ParMultiDo(StripIds)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.627Z: Fusing consumer Window.Into()2/Window.Assign into Collect start time metrics (co-input)/ParMultiDo(TimeMonitor)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.654Z: Fusing consumer CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) into Window.Into()2/Window.Assign
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.697Z: Fusing consumer CoGroupByKey/GBK/MergeBuckets into CoGroupByKey/GBK/ReadStream
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.733Z: Fusing consumer CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) into CoGroupByKey/GBK/MergeBuckets
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.755Z: Fusing consumer Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) into CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.787Z: Fusing consumer Collect total bytes/ParMultiDo(ByteMonitor) into Ungroup and reiterate/ParMultiDo(UngroupAndReiterate)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.820Z: Fusing consumer Collect end time metrics/ParMultiDo(TimeMonitor) into Collect total bytes/ParMultiDo(ByteMonitor)
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:24.993Z: Running job using Streaming Engine
May 06, 2022 12:36:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:25.292Z: Starting 5 ****s in us-central1-b...
May 06, 2022 12:36:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:35.180Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
May 06, 2022 12:36:48 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:36:47.859Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 06, 2022 12:37:49 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T12:37:47.668Z: Workers have started successfully.
May 06, 2022 4:01:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T16:01:11.959Z: Cancel request is committed for workflow job: 2022-05-06_05_36_09-16672906884944557169.
May 06, 2022 4:01:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T16:01:12.063Z: Cleaning up.
May 06, 2022 4:01:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T16:01:12.173Z: Stopping **** pool...
May 06, 2022 4:01:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T16:01:12.236Z: Stopping **** pool...
May 06, 2022 4:01:48 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T16:01:46.626Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
May 06, 2022 4:01:48 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-06T16:01:46.663Z: Worker pool stopped.
May 06, 2022 4:01:56 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2022-05-06_05_36_09-16672906884944557169 finished with status CANCELLED.
Load test results for test (ID): aea7067d-9935-4110-bf99-16a4bf6df520 and timestamp: 2022-05-06T12:35:56.758000000Z:
Metric: Value:
dataflow_v2_java17_runtime_sec 12160.557
dataflow_v2_java17_total_bytes_count 3.05775743E10
Exception in thread "main" java.lang.RuntimeException: Invalid job state: CANCELLED.
at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51)
at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.run(CoGroupByKeyLoadTest.java:62)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.main(CoGroupByKeyLoadTest.java:157)
> Task :sdks:java:testing:load-tests:run FAILED
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220506123132
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:df7a799483a8dfcf0e4964b95e5dc6ef838d5908578aa974f5608b4b2f79498f
Deleted: sha256:c61e2bf56473ec4563f54698aa2a8e107b9212d2f9cb4249e1b8926eebd47660
Deleted: sha256:9d9130c54f1baeb08b94a07964e9e22ecd4bab33862e7798a10319d1d8d4b08d
Deleted: sha256:8467a3fb70145cc93817e21ee5c868eec7532f4e86ca70e4d38f522af0cb398c
Deleted: sha256:59fbca7d507b996b1d51fe22b67984f3ba7e986f1c53f6e3a14639d6277f637d
Deleted: sha256:c5e291eab10e7283e0ca96918031f1a9e83e8b8411f7ac5b1f40ec2f30ba230e
Deleted: sha256:6f601d14f835a5e2a860e8838164b3f7b7d0e20a6c80cab2e040083a47b0b56f
Deleted: sha256:b495ad196c71a0563ffb223307fb959e64e64a20da8e12016bb4602bf19fa9a5
Deleted: sha256:7730c370c352c50337fe7f2bad2ce7e584bfef3be35d36987d4541fbb7c06a2d
Deleted: sha256:6ee922752cb80e19cfd4beb7f1111c74c090f8d970a315f577ca3ab541447ab4
Deleted: sha256:9e190c203486d33f72fe7036f544e381072667a81abf21b65a53927a8ca3ddd9
Deleted: sha256:f2bd20d129b02ee50849e5c9152aac79942dba3415537ea9cba062da73f12839
Deleted: sha256:2909b2457cefac16105342dbdf59f3e9288bd44393da551bd3665665ede39ac8
Deleted: sha256:00968e40a472f2b5ad0dc5d917c967cc8b28c86c62b3207ac0a6e3f943550041
Deleted: sha256:f4d6879504907af6782a76ee389504614fb7ab9455c4b107eb02f2c19046ed49
Deleted: sha256:4be40ff13744c31b7d2eb8e31c8e9e5dd9b9690ef2a3bec37134190d3c85cdb0
Deleted: sha256:36bcd0ee35e63597b3e643aef706aac4fbef299ba2773de693bc6ce03688f272
Deleted: sha256:da888297f316c3a48e5f9787b1fe1a9384df56871c0e907eeb0fd44a91012df7
Deleted: sha256:58d9461f9b85e6dc8dce3d9061ab1b7690a0a8b13fc305f246ef037183b06bf3
Deleted: sha256:9e3aed6009e7e4f7dbf0492e1853b2a2549a3e81cfd12daeaeabfb6094af94bf
Deleted: sha256:75fb1d5e10ded2a00433da41d70fbc05be0aa093ebdeb90611cba0b377d7fec6
Deleted: sha256:e999ffd0c7debe92dbcad4eabf1788922fefe6e251093e945a669a8f458b1c67
Deleted: sha256:9b05c11211514e7340a5e8839f29b37218653537f63497a47cede3c2125bd15b
Deleted: sha256:65e811058f16c4f2bfe8c6b1a1d910ea4dc0532e1a41c80eb57bbc9fded148cb
Deleted: sha256:b66d07df3cdafc664e5592122dc648cfb3250d33c251f4ec18d46206d483bb23
Deleted: sha256:1427b463374c90c14de7918ef0f0c7b26c7fd8a48ca5e4b9e3ef08203ad7c545
Deleted: sha256:d637282d3e86cd34969b59506e5e78e9bb4e3cde032e3c4f9e6e04c08a078941
Deleted: sha256:89a707572366f6809adb4362ae03d1c016718b4c4f45fe40192369f027375415
Deleted: sha256:4d36ad43cbe6758277403104e5f851d113c19fe8dd2e1cbd383416b8ac45d28d
Deleted: sha256:a7ce6b29ef0448179f5fc9bd58e3ff1eb05009c7417e54905a9dd6b2a10f7fff
Deleted: sha256:632d214ecd9402e629ff5d15ce8a4c318d85c083035e0043a09c7d41bdc02d30
Deleted: sha256:6eb47e9cb58a35946ffaab9ad85d0fbe80def98404dee173c9e679c55c36d57e
Deleted: sha256:affbfb0c7a73592504cf91a2cbde5d28f930f33acc93430d08460a0c68304935
Deleted: sha256:239810707ccfa57dd42da91f0a3261bf3902d2c487fa6c35dc7241f5e86b3335
Deleted: sha256:46e146925a73a269ca21a84f848012801e5f0e57e63c23c8dd5206c23efda7ec
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220506123132]
- referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:df7a799483a8dfcf0e4964b95e5dc6ef838d5908578aa974f5608b4b2f79498f]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220506123132] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:df7a799483a8dfcf0e4964b95e5dc6ef838d5908578aa974f5608b4b2f79498f])].
Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:df7a799483a8dfcf0e4964b95e5dc6ef838d5908578aa974f5608b4b2f79498f
Digests:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:df7a799483a8dfcf0e4964b95e5dc6ef838d5908578aa974f5608b4b2f79498f
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:df7a799483a8dfcf0e4964b95e5dc6ef838d5908578aa974f5608b4b2f79498f].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:load-tests:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 3h 31m 24s
110 actionable tasks: 73 executed, 33 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/l5ae3vt3byunc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17 #136
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/136/display/redirect?page=changes>
Changes:
[Heejong Lee] [BEAM-9245] Unable to pull datatore Entity which contains dict
[bulat.safiullin] [BEAM-14382] [Website] add banner container for with css, images, html
[Jan Lukavský] [BEAM-14196] add test verifying output watermark propagation in bundle
[Jan Lukavský] [BEAM-14196] Fix FlinkRunner mid-bundle output watermark handling
[nielm] [BEAM-14405] Fix NPE when ProjectID is not specified in a template
[bulat.safiullin] [BEAM-14382] change mobile banner img, add padding to banner section
[ahmedabualsaud] fix test decotrator typo
[noreply] Merge pull request #17440 from [BEAM-14329] Enable exponential backoff
[noreply] [BEAM-11104] Fix output forwarding issue for ProcessContinuations
[noreply] re-add testing package to pydoc (#17524)
[Heejong Lee] add test
[noreply] [BEAM-14250] Amended the workaround (#17531)
[noreply] [BEAM-11104] Fix broken split result validation (#17546)
[noreply] Fixed a SQL and screenshots in the Beam SQL blog (#17545)
[noreply] Merge pull request #17417: [BEAM-14388] Address some performance
[noreply] [BEAM-14386] [Flink] Support for scala 2.12 (#17512)
[noreply] [BEAM-14294] Worker changes to support trivial Batched DoFns (#17384)
[zyichi] Moving to 2.40.0-SNAPSHOT on master branch.
[noreply] [BEAM-14048] [CdapIO] Add ConfigWrapper for building CDAP PluginConfigs
------------------------------------------
[...truncated 50.60 KB...]
a037458de4e0: Preparing
a0aeee58f513: Waiting
d7b56d57c324: Waiting
04d5df9ec8e8: Waiting
dc197617b2ee: Waiting
bafdbe68e4ae: Preparing
c85968a0c042: Waiting
3263c0e06234: Waiting
3bc383470c05: Waiting
673012901f20: Waiting
e93827457889: Waiting
a13c519c6361: Preparing
bafdbe68e4ae: Waiting
a13c519c6361: Waiting
a5830f62fc2b: Waiting
a037458de4e0: Waiting
8996c988e5df: Pushed
882944b20e14: Pushed
c72f16974de9: Pushed
0d42ebd3c794: Pushed
911b1a8b5e57: Pushed
a0aeee58f513: Pushed
a81a04dbaea7: Pushed
d7b56d57c324: Pushed
1d4e401c8952: Pushed
04d5df9ec8e8: Pushed
a5830f62fc2b: Pushed
3bc383470c05: Layer already exists
e93827457889: Layer already exists
c85968a0c042: Pushed
08fa02ce37eb: Layer already exists
a037458de4e0: Layer already exists
bafdbe68e4ae: Layer already exists
a13c519c6361: Layer already exists
3263c0e06234: Pushed
dc197617b2ee: Pushed
673012901f20: Pushed
20220505123041: digest: sha256:131e4451715f7df49079705cc0d0df461d263e958b7b9a5e9d379950b22ba9a0 size: 4729
> Task :sdks:java:testing:load-tests:run
May 05, 2022 12:32:46 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 05, 2022 12:32:46 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 221 files. Enable logging at DEBUG level to see which files will be staged.
May 05, 2022 12:32:47 PM org.apache.beam.sdk.Pipeline validate
WARNING: The following transforms do not have stable unique names: Window.Into()
May 05, 2022 12:32:47 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
May 05, 2022 12:32:49 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 221 files from PipelineOptions.filesToStage to staging location to prepare for execution.
May 05, 2022 12:32:49 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 221 files cached, 0 files newly uploaded in 0 seconds
May 05, 2022 12:32:49 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/
May 05, 2022 12:32:50 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <119943 bytes, hash d2875933a8f7081a206ea4f4f77ec0d39d59892254cb85fcb163b4db2e98fae1> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-0odZM6j3CBogbqT0937A051ZiSJUy4X8sWO02y6Y-uE.pb
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1
May 05, 2022 12:32:51 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@463561c5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@659feb22, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3468ee6e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2f4b98f6, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@421def93, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@58c1da09, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2b2954e1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@58d6e55a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@751ae8a4, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@235d659c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4232b34a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2da16263, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@f5ce0bb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@47e51549, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@101a461c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@360e9c06, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5ebffb44, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@311ff287, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7377781e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@31db34da]
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/StripIds as step s2
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (input) as step s3
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s4
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s5
May 05, 2022 12:32:51 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3a2e9f5b, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7da34b26, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@10c47c79, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3b2553d9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@753fd7a1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@12a2585b, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@21b6c9c2, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@309cedb6, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3b95a6db, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bf54172, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2c9a6717, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7b3cde6f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6d091cad, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7c663eaf, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bb5ceb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1e692555, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3ba0ae41, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@76fe6cdc, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2ffb3aec, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@786ff1cb]
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/StripIds as step s6
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (co-input) as step s7
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()2/Window.Assign as step s8
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable0 as step s9
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable1 as step s10
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/Flatten as step s11
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/GBK as step s12
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s13
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate as step s14
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect total bytes as step s15
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics as step s16
May 05, 2022 12:32:51 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.40.0-SNAPSHOT
May 05, 2022 12:32:52 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-05_05_32_52-6914658191205046925?project=apache-beam-testing
May 05, 2022 12:32:52 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-05-05_05_32_52-6914658191205046925
May 05, 2022 12:32:52 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2022-05-05_05_32_52-6914658191205046925
May 05, 2022 12:33:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-05-05T12:33:04.755Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java170dataflow0v20streaming0cogbk01-jenkins-05-839r. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions
May 05, 2022 12:33:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:12.969Z: Worker configuration: e2-standard-2 in us-central1-b.
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:13.766Z: Expanding SplittableParDo operations into optimizable parts.
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:13.798Z: Expanding CollectionToSingleton operations into optimizable parts.
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:13.898Z: Expanding CoGroupByKey operations into optimizable parts.
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:13.959Z: Expanding SplittableProcessKeyed operations into optimizable parts.
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:13.987Z: Expanding GroupByKey operations into streaming Read/Write steps
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.068Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.146Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.178Z: Unzipping flatten CoGroupByKey-Flatten for input CoGroupByKey-MakeUnionTable0-ParMultiDo-ConstructUnionTable-.output
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.208Z: Fusing unzipped copy of CoGroupByKey/GBK/WriteStream, through flatten CoGroupByKey/Flatten, into producer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.232Z: Fusing consumer CoGroupByKey/GBK/WriteStream into CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.252Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.281Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.316Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.346Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.380Z: Fusing consumer Collect start time metrics (input)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.404Z: Fusing consumer Window.Into()/Window.Assign into Collect start time metrics (input)/ParMultiDo(TimeMonitor)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.461Z: Fusing consumer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) into Window.Into()/Window.Assign
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.500Z: Fusing consumer Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read co-input/Impulse
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.531Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.563Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.596Z: Fusing consumer Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.630Z: Fusing consumer Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) into Read co-input/ParDo(StripIds)/ParMultiDo(StripIds)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.665Z: Fusing consumer Window.Into()2/Window.Assign into Collect start time metrics (co-input)/ParMultiDo(TimeMonitor)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.697Z: Fusing consumer CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) into Window.Into()2/Window.Assign
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.718Z: Fusing consumer CoGroupByKey/GBK/MergeBuckets into CoGroupByKey/GBK/ReadStream
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.744Z: Fusing consumer CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) into CoGroupByKey/GBK/MergeBuckets
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.781Z: Fusing consumer Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) into CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.848Z: Fusing consumer Collect total bytes/ParMultiDo(ByteMonitor) into Ungroup and reiterate/ParMultiDo(UngroupAndReiterate)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:14.877Z: Fusing consumer Collect end time metrics/ParMultiDo(TimeMonitor) into Collect total bytes/ParMultiDo(ByteMonitor)
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:15.025Z: Running job using Streaming Engine
May 05, 2022 12:33:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:15.265Z: Starting 5 ****s in us-central1-b...
May 05, 2022 12:33:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:41.370Z: Autoscaling: Raised the number of ****s to 4 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 05, 2022 12:33:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:41.398Z: Resized **** pool to 4, though goal was 5. This could be a quota issue.
May 05, 2022 12:33:48 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:46.902Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
May 05, 2022 12:33:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:33:51.587Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 05, 2022 12:34:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T12:34:50.485Z: Workers have started successfully.
May 05, 2022 3:05:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T15:05:19.624Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
May 05, 2022 3:05:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T15:05:24.302Z: Worker configuration: e2-standard-2 in us-central1-b.
May 05, 2022 4:00:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T16:00:49.572Z: Cancel request is committed for workflow job: 2022-05-05_05_32_52-6914658191205046925.
May 05, 2022 4:00:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T16:00:49.600Z: Cleaning up.
May 05, 2022 4:00:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T16:00:49.808Z: Stopping **** pool...
May 05, 2022 4:00:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T16:00:49.897Z: Stopping **** pool...
May 05, 2022 4:01:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T16:01:26.536Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
May 05, 2022 4:01:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-05T16:01:26.569Z: Worker pool stopped.
May 05, 2022 4:01:39 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2022-05-05_05_32_52-6914658191205046925 finished with status CANCELLED.
Load test results for test (ID): 3e4bb4f5-730b-4386-9ddb-8239732ef044 and timestamp: 2022-05-05T12:32:46.775000000Z:
Metric: Value:
dataflow_v2_java17_runtime_sec 12316.972
dataflow_v2_java17_total_bytes_count 3.41388437E10
Exception in thread "main" java.lang.RuntimeException: Invalid job state: CANCELLED.
at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51)
at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.run(CoGroupByKeyLoadTest.java:62)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.main(CoGroupByKeyLoadTest.java:157)
> Task :sdks:java:testing:load-tests:run FAILED
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220505123041
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:131e4451715f7df49079705cc0d0df461d263e958b7b9a5e9d379950b22ba9a0
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220505123041]
- referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:131e4451715f7df49079705cc0d0df461d263e958b7b9a5e9d379950b22ba9a0]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220505123041] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:131e4451715f7df49079705cc0d0df461d263e958b7b9a5e9d379950b22ba9a0])].
Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:131e4451715f7df49079705cc0d0df461d263e958b7b9a5e9d379950b22ba9a0
Digests:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:131e4451715f7df49079705cc0d0df461d263e958b7b9a5e9d379950b22ba9a0
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:131e4451715f7df49079705cc0d0df461d263e958b7b9a5e9d379950b22ba9a0].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:load-tests:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 3h 31m 20s
110 actionable tasks: 76 executed, 30 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/vqaloxpdhzprm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17 #135
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_Java17/135/display/redirect?page=changes>
Changes:
[noreply] fix: JDBC config schema fields order
[Brian Hulette] Revert "Merge pull request #17255 from kileys/test-revert"
[Brian Hulette] BEAM-14231: bypass schema cache for
[noreply] [BEAM-13657] Follow up update version warning in __init__ (#17493)
[noreply] Merge pull request #17431 from [BEAM-14273] Add integration tests for BQ
[noreply] Merge pull request #17205 from [BEAM-14145] [Website] add carousel to
[noreply] [BEAM-14064] fix es io windowing (#17112)
[noreply] [BEAM-13670] Upgraded ipython from v7 to v8 (#17529)
[noreply] [BEAM-11104] Enable ProcessContinuation return values, add unit test
[Robert Bradshaw] [BEAM-14403] Allow Prime to be used with legacy workers.
[noreply] [BEAM-11106] Support drain in Go SDK (#17432)
[noreply] add __Init__ to inference. (#17514)
------------------------------------------
[...truncated 50.32 KB...]
3bc383470c05: Preparing
e93827457889: Preparing
08fa02ce37eb: Preparing
a037458de4e0: Preparing
bafdbe68e4ae: Preparing
a13c519c6361: Preparing
3bc70159237a: Waiting
a6e4d1804105: Waiting
2d543f6e6e21: Waiting
0307cef3be55: Waiting
a037458de4e0: Waiting
08fa02ce37eb: Waiting
8b92e4958fd1: Waiting
bafdbe68e4ae: Waiting
9969bcd1e3fd: Waiting
e93827457889: Waiting
3bc383470c05: Waiting
2520eb2e8567: Waiting
72bfc7510f27: Waiting
a4ad5e0bd783: Waiting
f116eab367a0: Pushed
331c9a0b820a: Pushed
cb7330ae032c: Pushed
25a6908e0e13: Pushed
b7bbab1ffc45: Pushed
a4ad5e0bd783: Pushed
3bc70159237a: Pushed
2520eb2e8567: Pushed
9745019c1f1b: Pushed
2d543f6e6e21: Pushed
72bfc7510f27: Pushed
0307cef3be55: Pushed
3bc383470c05: Layer already exists
e93827457889: Layer already exists
08fa02ce37eb: Layer already exists
a037458de4e0: Layer already exists
bafdbe68e4ae: Layer already exists
a13c519c6361: Layer already exists
8b92e4958fd1: Pushed
9969bcd1e3fd: Pushed
a6e4d1804105: Pushed
20220504123041: digest: sha256:b607c702456435e49fbe29707a5e8a335df87c310c8eae2069a24e6102daeb1f size: 4729
> Task :sdks:java:testing:load-tests:run
May 04, 2022 12:32:36 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 04, 2022 12:32:37 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 221 files. Enable logging at DEBUG level to see which files will be staged.
May 04, 2022 12:32:38 PM org.apache.beam.sdk.Pipeline validate
WARNING: The following transforms do not have stable unique names: Window.Into()
May 04, 2022 12:32:38 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
May 04, 2022 12:32:40 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 221 files from PipelineOptions.filesToStage to staging location to prepare for execution.
May 04, 2022 12:32:41 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 221 files cached, 0 files newly uploaded in 0 seconds
May 04, 2022 12:32:41 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-perf-tests/loadtests/staging/
May 04, 2022 12:32:41 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <119943 bytes, hash a19c96ecdc20c9355bed3a3f2f46e859fa201abbc15dd86e5cf410cc6d045525> to gs://temp-storage-for-perf-tests/loadtests/staging/pipeline-oZyW7NwgyTVb7To_L0boWfogGrvBXdhuXPQQzG0EVSU.pb
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s1
May 04, 2022 12:32:43 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@659feb22, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3468ee6e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2f4b98f6, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@421def93, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@58c1da09, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2b2954e1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@58d6e55a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@751ae8a4, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@235d659c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4232b34a, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2da16263, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@f5ce0bb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@47e51549, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@101a461c, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@360e9c06, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5ebffb44, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@311ff287, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7377781e, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@31db34da, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@109f8c7e]
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read input/StripIds as step s2
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (input) as step s3
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()/Window.Assign as step s4
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/DataflowRunner.StreamingUnboundedRead.ReadWithIds as step s5
May 04, 2022 12:32:43 PM org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource split
INFO: Split into 20 bundles of sizes: [org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bf54172, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2c9a6717, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7b3cde6f, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6d091cad, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@7c663eaf, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3bb5ceb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@1e692555, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@3ba0ae41, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@76fe6cdc, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@2ffb3aec, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@786ff1cb, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@46039a21, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@431e86b1, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@35c4e864, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@32a2a6be, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@682af059, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5f36c8e3, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@4da39ca9, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@6a9344f5, org.apache.beam.sdk.io.synthetic.SyntheticUnboundedSource@5584d9c6]
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Read co-input/StripIds as step s6
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect start time metrics (co-input) as step s7
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Window.Into()2/Window.Assign as step s8
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable0 as step s9
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/MakeUnionTable1 as step s10
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/Flatten as step s11
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/GBK as step s12
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CoGroupByKey/ConstructCoGbkResultFn as step s13
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Ungroup and reiterate as step s14
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect total bytes as step s15
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Collect end time metrics as step s16
May 04, 2022 12:32:43 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.39.0-SNAPSHOT
May 04, 2022 12:32:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2022-05-04_05_32_43-7390128921444977935?project=apache-beam-testing
May 04, 2022 12:32:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2022-05-04_05_32_43-7390128921444977935
May 04, 2022 12:32:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2022-05-04_05_32_43-7390128921444977935
May 04, 2022 12:32:48 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2022-05-04T12:32:47.612Z: The workflow name is not a valid Cloud Label. Labels applied to Cloud resources (such as GCE Instances) for monitoring will be labeled with this modified job name: load0tests0java170dataflow0v20streaming0cogbk01-jenkins-05-q8mb. For the best monitoring experience, please name your job with a valid Cloud Label. For details, see: https://cloud.google.com/compute/docs/labeling-resources#restrictions
May 04, 2022 12:32:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:52.883Z: Worker configuration: e2-standard-2 in us-central1-b.
May 04, 2022 12:32:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:53.521Z: Expanding SplittableParDo operations into optimizable parts.
May 04, 2022 12:32:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:53.565Z: Expanding CollectionToSingleton operations into optimizable parts.
May 04, 2022 12:32:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:53.692Z: Expanding CoGroupByKey operations into optimizable parts.
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:53.790Z: Expanding SplittableProcessKeyed operations into optimizable parts.
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:53.843Z: Expanding GroupByKey operations into streaming Read/Write steps
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:53.962Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.105Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.144Z: Unzipping flatten CoGroupByKey-Flatten for input CoGroupByKey-MakeUnionTable0-ParMultiDo-ConstructUnionTable-.output
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.191Z: Fusing unzipped copy of CoGroupByKey/GBK/WriteStream, through flatten CoGroupByKey/Flatten, into producer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.237Z: Fusing consumer CoGroupByKey/GBK/WriteStream into CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.311Z: Fusing consumer Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read input/Impulse
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.350Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.403Z: Fusing consumer Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.441Z: Fusing consumer Read input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.483Z: Fusing consumer Collect start time metrics (input)/ParMultiDo(TimeMonitor) into Read input/ParDo(StripIds)/ParMultiDo(StripIds)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.527Z: Fusing consumer Window.Into()/Window.Assign into Collect start time metrics (input)/ParMultiDo(TimeMonitor)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.581Z: Fusing consumer CoGroupByKey/MakeUnionTable0/ParMultiDo(ConstructUnionTable) into Window.Into()/Window.Assign
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.620Z: Fusing consumer Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource) into Read co-input/Impulse
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.659Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction into Read co-input/ParDo(OutputSingleSource)/ParMultiDo(OutputSingleSource)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.705Z: Fusing consumer Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/SplitWithSizing into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/PairWithRestriction
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.742Z: Fusing consumer Read co-input/ParDo(StripIds)/ParMultiDo(StripIds) into Read-co-input-ParDo-UnboundedSourceAsSDFWrapper--ParMultiDo-UnboundedSourceAsSDFWrapper-/ProcessElementAndRestrictionWithSizing
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.779Z: Fusing consumer Collect start time metrics (co-input)/ParMultiDo(TimeMonitor) into Read co-input/ParDo(StripIds)/ParMultiDo(StripIds)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.819Z: Fusing consumer Window.Into()2/Window.Assign into Collect start time metrics (co-input)/ParMultiDo(TimeMonitor)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.858Z: Fusing consumer CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable) into Window.Into()2/Window.Assign
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.925Z: Fusing consumer CoGroupByKey/GBK/MergeBuckets into CoGroupByKey/GBK/ReadStream
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:54.966Z: Fusing consumer CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult) into CoGroupByKey/GBK/MergeBuckets
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:55.004Z: Fusing consumer Ungroup and reiterate/ParMultiDo(UngroupAndReiterate) into CoGroupByKey/ConstructCoGbkResultFn/ParMultiDo(ConstructCoGbkResult)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:55.043Z: Fusing consumer Collect total bytes/ParMultiDo(ByteMonitor) into Ungroup and reiterate/ParMultiDo(UngroupAndReiterate)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:55.123Z: Fusing consumer Collect end time metrics/ParMultiDo(TimeMonitor) into Collect total bytes/ParMultiDo(ByteMonitor)
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:55.382Z: Running job using Streaming Engine
May 04, 2022 12:32:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:32:55.912Z: Starting 5 ****s in us-central1-b...
May 04, 2022 12:33:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:33:17.504Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
May 04, 2022 12:33:22 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:33:20.346Z: Autoscaling: Raised the number of ****s to 4 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 04, 2022 12:33:22 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:33:20.377Z: Resized **** pool to 4, though goal was 5. This could be a quota issue.
May 04, 2022 12:33:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:33:30.625Z: Autoscaling: Raised the number of ****s to 5 so that the pipeline can catch up with its backlog and keep up with its input rate.
May 04, 2022 12:34:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T12:34:22.731Z: Workers have started successfully.
May 04, 2022 4:01:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T16:00:59.996Z: Cancel request is committed for workflow job: 2022-05-04_05_32_43-7390128921444977935.
May 04, 2022 4:01:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T16:01:00.111Z: Cleaning up.
May 04, 2022 4:01:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T16:01:00.314Z: Stopping **** pool...
May 04, 2022 4:01:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T16:01:00.383Z: Stopping **** pool...
May 04, 2022 4:01:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T16:01:35.915Z: Autoscaling: Reduced the number of ****s to 0 based on low average **** CPU utilization, and the pipeline having sufficiently low backlog and keeping up with input rate.
May 04, 2022 4:01:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2022-05-04T16:01:35.987Z: Worker pool stopped.
May 04, 2022 4:01:44 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2022-05-04_05_32_43-7390128921444977935 finished with status CANCELLED.
Load test results for test (ID): 1fb5cbe1-adcf-4c95-8641-b4766f1d23bb and timestamp: 2022-05-04T12:32:37.757000000Z:
Metric: Value:
dataflow_v2_java17_runtime_sec 12363.386
dataflow_v2_java17_total_bytes_count 4.12943456E10
Exception in thread "main" java.lang.RuntimeException: Invalid job state: CANCELLED.
at org.apache.beam.sdk.loadtests.JobFailure.handleFailure(JobFailure.java:51)
at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:139)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.run(CoGroupByKeyLoadTest.java:62)
at org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest.main(CoGroupByKeyLoadTest.java:157)
> Task :sdks:java:testing:load-tests:run FAILED
> Task :runners:google-cloud-dataflow-java:cleanUpDockerJavaImages
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220504123041
Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b607c702456435e49fbe29707a5e8a335df87c310c8eae2069a24e6102daeb1f
Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220504123041]
- referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b607c702456435e49fbe29707a5e8a335df87c310c8eae2069a24e6102daeb1f]
Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20220504123041] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b607c702456435e49fbe29707a5e8a335df87c310c8eae2069a24e6102daeb1f])].
Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b607c702456435e49fbe29707a5e8a335df87c310c8eae2069a24e6102daeb1f
Digests:
- us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b607c702456435e49fbe29707a5e8a335df87c310c8eae2069a24e6102daeb1f
Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:b607c702456435e49fbe29707a5e8a335df87c310c8eae2069a24e6102daeb1f].
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:java:testing:load-tests:run'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 3h 31m 26s
110 actionable tasks: 73 executed, 33 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/d7u3qhbfqkica
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org