You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2021/09/09 12:07:00 UTC

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #1486

See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1486/display/redirect?page=changes>

Changes:

[suztomo] [BEAM-11205] Upgrading the Libraries BOM to v22

[randomstep] [BEAM-12708] Bump arrow-memory-netty

[Etienne Chauchot] [BEAM-12153] implement GroupByKey with CombinePerKey with Concatenate

[Etienne Chauchot] [BEAM-11023] Increase memory in SS Validates runner tests to avoid OOM

[vincent.marquez] [BEAM-9008] adds CassandraIO.readAll

[Etienne Chauchot] [BEAM-12727] extract Concatenate CombineFn to runner-core module to

[noreply] Add display data for JdbcIO.write (#15460)

[noreply] Merge pull request #15480: [BEAM-12356] Make sure DatasetService is

[noreply] [BEAM-11981] Java Bigtable - Implement IO Request Count metrics (#15342)

[noreply] [BEAM-12834] Improve Go SDK cross-language documentation and API.


------------------------------------------
[...truncated 2.67 MB...]
Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:01:48.466Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign
Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:01:48.499Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:01:48.534Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify
Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:01:48.584Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read
Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:01:48.636Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow
Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:01:48.674Z: Fusing consumer WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten
Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:01:49.102Z: Executing operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Create
Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:01:49.167Z: Starting 1 workers in us-central1-c...
Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:01:49.496Z: Finished operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Create
Sep 09, 2021 12:01:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:01:49.706Z: Executing operation TextIO.Read/Read+ParseGameEvent+ExtractUserScore/MapElements/Map+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Reify+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Write
Sep 09, 2021 12:02:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:02:07.919Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Sep 09, 2021 12:02:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:02:25.677Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s).
Sep 09, 2021 12:03:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:02.846Z: Workers have started successfully.
Sep 09, 2021 12:03:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:02.884Z: Workers have started successfully.
Sep 09, 2021 12:03:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:22.989Z: Finished operation TextIO.Read/Read+ParseGameEvent+ExtractUserScore/MapElements/Map+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Reify+ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Write
Sep 09, 2021 12:03:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:23.059Z: Executing operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Close
Sep 09, 2021 12:03:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:23.127Z: Finished operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Close
Sep 09, 2021 12:03:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:23.214Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
Sep 09, 2021 12:03:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:24.499Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
Sep 09, 2021 12:03:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:24.677Z: Executing operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Read+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract+WriteUserScoreSums/ConvertToRow+WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
Sep 09, 2021 12:03:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:28.371Z: Finished operation ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/Read+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues+ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract+WriteUserScoreSums/ConvertToRow+WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
Sep 09, 2021 12:03:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:28.543Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close
Sep 09, 2021 12:03:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:28.641Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close
Sep 09, 2021 12:03:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:28.713Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:31.720Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:31.905Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1
Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:32.004Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1
Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:32.165Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView
Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:32.222Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView
Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:32.372Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:32.562Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Sep 09, 2021 12:03:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:32.746Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:35.860Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:35.925Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:35.979Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:36.051Z: Executing operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:36.333Z: Finished operation WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
Sep 09, 2021 12:03:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:36.484Z: Cleaning up.
Sep 09, 2021 12:03:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:03:36.590Z: Stopping worker pool...
Sep 09, 2021 12:05:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:05:42.708Z: Autoscaling: Resized worker pool from 1 to 0.
Sep 09, 2021 12:05:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:05:42.745Z: Worker pool stopped.
Sep 09, 2021 12:06:18 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2021-09-09_05_01_09-10897150730585566451 finished with status DONE.
gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt* | grep user19_BananaWallaby
total_score: 231, user: user19_BananaWallaby
Verified total_score: 231, user: user19_BananaWallaby

**************************************
* SUCCESS: UserScore successfully run on DataflowRunner.
**************************************

gsutil rm gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt*
Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt-00000-of-00001...
/ [1 objects]                                                                   
Operation completed over 1 objects.                                              

**************************************
* Test: Running: HourlyTeamScore example on DataflowRunner
**************************************

mvn compile exec:java -q       -Dmaven.wagon.http.retryHandler.class=default       -Dmaven.wagon.http.retryHandler.count=5       -Dmaven.wagon.http.pool=false       -Dmaven.wagon.httpconnectionManager.ttlSeconds=120       -Dhttp.keepAlive=false       -Dexec.mainClass=org.apache.beam.examples.complete.game.HourlyTeamScore       -Dexec.args="--tempLocation=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp --runner=DataflowRunner --input=gs://dataflow-samples/game/5000_gaming_data.csv --project=apache-beam-testing --region=us-central1 --output=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt "       -Pdataflow-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.4
Sep 09, 2021 12:06:25 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
Sep 09, 2021 12:06:26 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 228 files. Enable logging at DEBUG level to see which files will be staged.
Sep 09, 2021 12:06:26 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
Sep 09, 2021 12:06:28 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/
Sep 09, 2021 12:06:28 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <235364 bytes, hash b0aea769493c0e3ef1ef0c2ab533ac2ad952c1c273e9f5c3b008f4b857bb9fed> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-sK6naUk8Dj7x7wwqtTOsKtlSwcJz6fXDsAj0uFe7n-0.pb
Sep 09, 2021 12:06:30 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 228 files from PipelineOptions.filesToStage to staging location to prepare for execution.
Sep 09, 2021 12:06:30 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 228 files cached, 0 files newly uploaded in 0 seconds
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding TextIO.Read/Read as step s1
Sep 09, 2021 12:06:31 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes
INFO: Filepattern gs://dataflow-samples/game/5000_gaming_data.csv matched 1 files with total size 386400
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ParseGameEvent as step s2
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding FilterStartTime/ParDo(Anonymous) as step s3
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding FilterEndTime/ParDo(Anonymous) as step s4
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding AddEventTimestamps/AddTimestamps as step s5
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding FixedWindowsTeam/Window.Assign as step s6
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractTeamScore/MapElements/Map as step s7
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey as step s8
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues as step s9
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/ConvertToRow as step s10
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey as step s11
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards as step s12
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles as step s13
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void key/AddKeys/Map as step s14
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign as step s15
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey as step s16
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable as step s17
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop key/Values/Map as step s18
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather bundles as step s19
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair with random key as step s20
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s21
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s22
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s23
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map as step s24
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize as step s25
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s26
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s27
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s28
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s29
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s30
Sep 09, 2021 12:06:31 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.34.0-SNAPSHOT
Sep 09, 2021 12:06:32 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-09_05_06_31-3524899216147049663?project=apache-beam-testing
Sep 09, 2021 12:06:32 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2021-09-09_05_06_31-3524899216147049663
Sep 09, 2021 12:06:32 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2021-09-09_05_06_31-3524899216147049663
Sep 09, 2021 12:06:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:06:35.161Z: Autoscaling is enabled for job 2021-09-09_05_06_31-3524899216147049663. The number of workers will be between 1 and 1000.
Sep 09, 2021 12:06:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:06:35.286Z: Autoscaling was automatically enabled for job 2021-09-09_05_06_31-3524899216147049663.
Sep 09, 2021 12:06:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:06:50.309Z: Worker configuration: e2-standard-2 in us-central1-c.
Sep 09, 2021 12:06:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2021-09-09T12:06:51.119Z: Workflow failed. Causes: Project apache-beam-testing has insufficient quota(s) to execute this workflow with 1 instances in region us-central1. Quota summary (required/available): 1/19451 instances, 2/0 CPUs, 25/260276 disk GB, 0/2397 SSD disk GB, 1/222 instance groups, 1/226 managed instance groups, 1/447 instance templates, 1/680 in-use IP addresses.

Please see https://cloud.google.com/compute/docs/resource-quotas about requesting more quota.
Sep 09, 2021 12:06:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:06:51.169Z: Cleaning up.
Sep 09, 2021 12:06:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:06:51.238Z: Worker pool stopped.
Sep 09, 2021 12:06:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-09T12:06:52.554Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Sep 09, 2021 12:06:56 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2021-09-09_05_06_31-3524899216147049663 failed with status FAILED.
gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt* | grep AzureBilby 
CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt*
CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt*
[ERROR] Failed command

> Task :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow FAILED

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1h 6m 38s
9 actionable tasks: 8 executed, 1 from cache

Publishing build scan...
https://gradle.com/s/j6xgujcv5qvde

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #1491

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1491/display/redirect?page=changes>


---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #1490

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1490/display/redirect>

Changes:


------------------------------------------
[...truncated 2.18 MB...]
INFO: Adding WordCount.CountWords/ParDo(ExtractWords) as step s2
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WordCount.CountWords/Count.PerElement/Init/Map as step s3
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey as step s4
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues as step s5
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding MapElements/Map as step s6
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign as step s7
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles as step s8
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step s9
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step s10
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum as step s11
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as step s12
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) as step s13
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView as step s14
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) as step s15
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) as step s16
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map as step s17
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize as step s18
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s19
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s20
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s21
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s22
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s23
Sep 13, 2021 11:17:16 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.34.0-SNAPSHOT
Sep 13, 2021 11:17:17 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-13_04_17_16-7721088385811039723?project=apache-beam-testing
Sep 13, 2021 11:17:17 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2021-09-13_04_17_16-7721088385811039723
Sep 13, 2021 11:17:17 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2021-09-13_04_17_16-7721088385811039723
Sep 13, 2021 11:17:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:20.382Z: Autoscaling is enabled for job 2021-09-13_04_17_16-7721088385811039723. The number of workers will be between 1 and 1000.
Sep 13, 2021 11:17:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:20.488Z: Autoscaling was automatically enabled for job 2021-09-13_04_17_16-7721088385811039723.
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:29.063Z: Worker configuration: e2-standard-2 in us-central1-c.
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:29.886Z: Expanding CoGroupByKey operations into optimizable parts.
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.020Z: Expanding GroupByKey operations into optimizable parts.
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.043Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.256Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.317Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.445Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.564Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.615Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key into WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.636Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.669Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.703Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.725Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.769Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.802Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.838Z: Unzipping flatten s12 for input s11.org.apache.beam.sdk.values.PCollection.<init>:408#5f2ef1f005ae0b4
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.868Z: Fusing unzipped copy of WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow), through flatten WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections, into producer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.901Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.937Z: Fusing consumer WordCount.CountWords/ParDo(ExtractWords) into ReadLines/Read
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:30.972Z: Fusing consumer WordCount.CountWords/Count.PerElement/Init/Map into WordCount.CountWords/ParDo(ExtractWords)
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.002Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial into WordCount.CountWords/Count.PerElement/Init/Map
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.039Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.080Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.116Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.140Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues
Sep 13, 2021 11:17:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.163Z: Fusing consumer MapElements/Map into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract
Sep 13, 2021 11:17:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.199Z: Fusing consumer WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign into MapElements/Map
Sep 13, 2021 11:17:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.226Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign
Sep 13, 2021 11:17:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.263Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Sep 13, 2021 11:17:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.294Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify
Sep 13, 2021 11:17:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.320Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read
Sep 13, 2021 11:17:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.360Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow
Sep 13, 2021 11:17:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.401Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten
Sep 13, 2021 11:17:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.796Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create
Sep 13, 2021 11:17:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:31.860Z: Starting 1 workers in us-central1-c...
Sep 13, 2021 11:17:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:35.468Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Sep 13, 2021 11:17:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:42.303Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create
Sep 13, 2021 11:17:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:17:42.461Z: Executing operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write
Sep 13, 2021 11:18:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:18:15.999Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running stage(s).
Sep 13, 2021 11:18:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:18:45.379Z: Workers have started successfully.
Sep 13, 2021 11:18:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:18:45.445Z: Workers have started successfully.
Sep 13, 2021 11:19:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:34.055Z: Finished operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write
Sep 13, 2021 11:19:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:34.122Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close
Sep 13, 2021 11:19:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:34.189Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close
Sep 13, 2021 11:19:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:34.253Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
Sep 13, 2021 11:19:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:34.455Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create
Sep 13, 2021 11:19:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:34.609Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
Sep 13, 2021 11:19:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:39.544Z: Finished operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
Sep 13, 2021 11:19:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:39.861Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close
Sep 13, 2021 11:19:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:39.928Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close
Sep 13, 2021 11:19:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:39.994Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
Sep 13, 2021 11:19:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:41.418Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
Sep 13, 2021 11:19:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:41.575Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1
Sep 13, 2021 11:19:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:41.685Z: Finished operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections/Unzipped-1
Sep 13, 2021 11:19:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:41.836Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView
Sep 13, 2021 11:19:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:41.911Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView
Sep 13, 2021 11:19:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:42.065Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Sep 13, 2021 11:19:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:42.270Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create
Sep 13, 2021 11:19:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:42.421Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Sep 13, 2021 11:19:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:43.405Z: Finished operation WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
Sep 13, 2021 11:19:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:43.482Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Sep 13, 2021 11:19:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:43.675Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close
Sep 13, 2021 11:19:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:43.740Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
Sep 13, 2021 11:19:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:44.002Z: Finished operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
Sep 13, 2021 11:19:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:44.164Z: Cleaning up.
Sep 13, 2021 11:19:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:19:44.254Z: Stopping worker pool...
Sep 13, 2021 11:21:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:21:58.970Z: Autoscaling: Resized worker pool from 1 to 0.
Sep 13, 2021 11:21:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-13T11:21:59.017Z: Worker pool stopped.
Sep 13, 2021 11:22:34 AM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2021-09-13_04_17_16-7721088385811039723 finished with status DONE.
gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* | grep Montague:
Montague: 47
Verified Montague: 47
gsutil rm gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count*
Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00000-of-00001...
/ [1 objects]                                                                   
Operation completed over 1 objects.                                              
[SUCCESS]

FAILURE: Build completed with 3 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 22m 17s
9 actionable tasks: 8 executed, 1 from cache

Publishing build scan...
https://gradle.com/s/c6hjhrns7bpsk

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #1489

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1489/display/redirect>

Changes:


------------------------------------------
[...truncated 2.27 MB...]
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-aether-provider/3.0/maven-aether-provider-3.0.jar (51 kB at 180 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-util/1.7/aether-util-1.7.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-impl/1.7/aether-impl-1.7.jar (106 kB at 351 kB/s)
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-spi/1.7/aether-spi-1.7.jar (14 kB at 45 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.2.3/plexus-classworlds-2.2.3.jar
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-model-builder/3.0/maven-model-builder-3.0.jar (148 kB at 482 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-api/1.7/aether-api-1.7.jar (74 kB at 239 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.0/maven-model-3.0.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-util/1.7/aether-util-1.7.jar (108 kB at 346 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/3.0/maven-settings-3.0.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar (29 kB at 87 kB/s)
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.2.3/plexus-classworlds-2.2.3.jar (46 kB at 140 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-invoker/3.0.1/maven-invoker-3.0.1.jar
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-artifact-transfer/0.12.0/maven-artifact-transfer-0.12.0.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-settings/3.0/maven-settings-3.0.jar (47 kB at 138 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-common-artifact-filters/3.0.1/maven-common-artifact-filters-3.0.1.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.0/maven-model-3.0.jar (165 kB at 486 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.11/commons-codec-1.11.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar (13 kB at 39 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-invoker/3.0.1/maven-invoker-3.0.1.jar (33 kB at 93 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-artifact-transfer/0.12.0/maven-artifact-transfer-0.12.0.jar (120 kB at 337 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-script-interpreter/1.0/maven-script-interpreter-1.0.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-common-artifact-filters/3.0.1/maven-common-artifact-filters-3.0.1.jar (61 kB at 169 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/beanshell/bsh/2.0b4/bsh-2.0b4.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar (26 kB at 71 kB/s)
[INFO] Downloading from central: https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.1/ant-1.8.1.jar
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.11/commons-codec-1.11.jar (335 kB at 903 kB/s)
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-script-interpreter/1.0/maven-script-interpreter-1.0.jar (21 kB at 54 kB/s)
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar (575 kB at 1.5 MB/s)
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/beanshell/bsh/2.0b4/bsh-2.0b4.jar (282 kB at 715 kB/s)
[INFO] Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.1/ant-1.8.1.jar (1.5 MB at 3.5 MB/s)
[INFO] Generating project in Batch mode
[INFO] Archetype repository not defined. Using the one from [org.apache.beam:beam-sdks-java-maven-archetypes-examples:2.24.0] found in catalog remote
[INFO] Downloading from test.release: https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-maven-archetypes-examples/2.34.0-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded from test.release: https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-maven-archetypes-examples/2.34.0-SNAPSHOT/maven-metadata.xml (1.7 kB at 2.7 kB/s)
[INFO] Downloading from test.release: https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-maven-archetypes-examples/2.34.0-SNAPSHOT/beam-sdks-java-maven-archetypes-examples-2.34.0-20210829.070151-5.pom
[INFO] Downloaded from test.release: https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-maven-archetypes-examples/2.34.0-SNAPSHOT/beam-sdks-java-maven-archetypes-examples-2.34.0-20210829.070151-5.pom (4.1 kB at 17 kB/s)
[INFO] Downloading from test.release: https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-maven-archetypes-examples/2.34.0-SNAPSHOT/beam-sdks-java-maven-archetypes-examples-2.34.0-20210829.070151-5.jar
[INFO] Downloaded from test.release: https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-maven-archetypes-examples/2.34.0-SNAPSHOT/beam-sdks-java-maven-archetypes-examples-2.34.0-20210829.070151-5.jar (224 kB at 326 kB/s)
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: beam-sdks-java-maven-archetypes-examples:2.34.0-SNAPSHOT
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: org.example
[INFO] Parameter: artifactId, Value: word-count-beam
[INFO] Parameter: version, Value: 0.1
[INFO] Parameter: package, Value: org.apache.beam.examples
[INFO] Parameter: packageInPathFormat, Value: org/apache/beam/examples
[INFO] Parameter: package, Value: org.apache.beam.examples
[INFO] Parameter: version, Value: 0.1
[INFO] Parameter: groupId, Value: org.example
[INFO] Parameter: targetPlatform, Value: 1.8
[INFO] Parameter: artifactId, Value: word-count-beam
[INFO] Project created from Archetype in dir: /tmp/groovy-generated-1517256378032535609-tmpdir/word-count-beam
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.222 s
[INFO] Finished at: 2021-09-12T11:42:20Z
[INFO] ------------------------------------------------------------------------
Verified [INFO] BUILD SUCCESS
cd word-count-beam
ls
pom.xml
src
Verified pom.xml
Verified src
ls src/main/java/org/apache/beam/examples/
common
complete
DebuggingWordCount.java
MinimalWordCount.java
subprocess
WindowedWordCount.java
WordCount.java
Verified WordCount.java
ls src/main/java/org/apache/beam/examples/complete/game/
GameStats.java
HourlyTeamScore.java
injector
LeaderBoard.java
StatefulTeamScore.java
UserScore.java
utils
Verified UserScore.java

**************************************
* Test: Running: UserScore example on DataflowRunner
**************************************

mvn compile exec:java -q       -Dmaven.wagon.http.retryHandler.class=default       -Dmaven.wagon.http.retryHandler.count=5       -Dmaven.wagon.http.pool=false       -Dmaven.wagon.httpconnectionManager.ttlSeconds=120       -Dhttp.keepAlive=false       -Dexec.mainClass=org.apache.beam.examples.complete.game.UserScore       -Dexec.args="--tempLocation=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp --runner=DataflowRunner --input=gs://dataflow-samples/game/5000_gaming_data.csv --project=apache-beam-testing --region=us-central1 --output=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt "       -Pdataflow-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.4
Sep 12, 2021 11:47:21 AM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
Sep 12, 2021 11:47:21 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 228 files. Enable logging at DEBUG level to see which files will be staged.
Sep 12, 2021 11:47:22 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
Sep 12, 2021 11:47:24 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/
Sep 12, 2021 11:47:24 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <186420 bytes, hash df4ab4cf26510c8dc2afd26b08bd31ac34d6b1ccf5c40abe9cd3dd361b77ddc7> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-30q0zyZRDI3Cr9JrCL0xrDTWscz1xAq-nNPdNht33cc.pb
Sep 12, 2021 11:47:25 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 228 files from PipelineOptions.filesToStage to staging location to prepare for execution.
Sep 12, 2021 11:47:25 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading /tmp/classes8926808956976425813.zip to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/classes-3cd6PSYvZUuQz-3WeSCOWZcMowFAKuzfpDPMNUYVbnE.jar
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 227 files cached, 1 files newly uploaded in 0 seconds
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding TextIO.Read/Read as step s1
Sep 12, 2021 11:47:26 AM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes
INFO: Filepattern gs://dataflow-samples/game/5000_gaming_data.csv matched 1 files with total size 386400
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ParseGameEvent as step s2
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractUserScore/MapElements/Map as step s3
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey as step s4
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues as step s5
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/ConvertToRow as step s6
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/RewindowIntoGlobal/Window.Assign as step s7
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles as step s8
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step s9
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step s10
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum as step s11
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as step s12
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) as step s13
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView as step s14
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) as step s15
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) as step s16
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map as step s17
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize as step s18
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s19
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s20
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s21
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s22
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s23
Sep 12, 2021 11:47:26 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.34.0-SNAPSHOT
Sep 12, 2021 11:47:27 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-12_04_47_26-13573532213139101580?project=apache-beam-testing
Sep 12, 2021 11:47:27 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2021-09-12_04_47_26-13573532213139101580
Sep 12, 2021 11:47:27 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2021-09-12_04_47_26-13573532213139101580
Sep 12, 2021 11:47:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-12T11:47:30.502Z: Autoscaling is enabled for job 2021-09-12_04_47_26-13573532213139101580. The number of workers will be between 1 and 1000.
Sep 12, 2021 11:47:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-12T11:47:30.608Z: Autoscaling was automatically enabled for job 2021-09-12_04_47_26-13573532213139101580.
Sep 12, 2021 11:47:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-12T11:47:38.589Z: Worker configuration: e2-standard-2 in us-central1-c.
Sep 12, 2021 11:47:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2021-09-12T11:47:39.463Z: Workflow failed. Causes: Project apache-beam-testing has insufficient quota(s) to execute this workflow with 1 instances in region us-central1. Quota summary (required/available): 1/24379 instances, 2/0 CPUs, 25/186866 disk GB, 0/2397 SSD disk GB, 1/285 instance groups, 1/288 managed instance groups, 1/514 instance templates, 1/608 in-use IP addresses.

Please see https://cloud.google.com/compute/docs/resource-quotas about requesting more quota.
Sep 12, 2021 11:47:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-12T11:47:39.509Z: Cleaning up.
Sep 12, 2021 11:47:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-12T11:47:39.577Z: Worker pool stopped.
Sep 12, 2021 11:47:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-12T11:47:40.751Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Sep 12, 2021 11:47:44 AM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2021-09-12_04_47_26-13573532213139101580 failed with status FAILED.
gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt* | grep user19_BananaWallaby
CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt*
CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt*
[ERROR] Failed command

> Task :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow FAILED

FAILURE: Build completed with 4 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflowBom'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:runQuickstartJavaDataflow'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

4: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 47m 22s
9 actionable tasks: 8 executed, 1 from cache

Publishing build scan...
https://gradle.com/s/hwbsa4n7ddk7q

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #1488

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1488/display/redirect?page=changes>

Changes:

[noreply] Register MapCoder, some comments/cleanup. (#15471)

[noreply] [BEAM-12588] Multimap user state proto changes (#15473)

[noreply] Added type annotations to some combiners missing it. (#15414)

[noreply] [BEAM-12634] JmsIO auto scaling feature (#15464)

[noreply] [BEAM-12662] Get Flink version from cluster. (#15223)

[noreply] Port changes from Pub/Sub Lite to beam (#15418)

[heejong] [BEAM-12805] Fix XLang CombinePerKey test by explicitly assigning the

[BenWhitehead] [BEAM-8376] Google Cloud Firestore Connector - Add handling for

[noreply] Decreasing peak memory usage for beam.TupleCombineFn (#15494)

[noreply] [BEAM-12802] Add support for prefetch through data layers down through

[noreply] [BEAM-11097] Add implementation of side input cache (#15483)


------------------------------------------
[...truncated 2.64 MB...]

**************************************
* SUCCESS: UserScore successfully run on DataflowRunner.
**************************************

gsutil rm gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt*
Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-userscore-result-dataflow-runner.txt-00000-of-00001...
/ [1 objects]                                                                   
Operation completed over 1 objects.                                              

**************************************
* Test: Running: HourlyTeamScore example on DataflowRunner
**************************************

mvn compile exec:java -q       -Dmaven.wagon.http.retryHandler.class=default       -Dmaven.wagon.http.retryHandler.count=5       -Dmaven.wagon.http.pool=false       -Dmaven.wagon.httpconnectionManager.ttlSeconds=120       -Dhttp.keepAlive=false       -Dexec.mainClass=org.apache.beam.examples.complete.game.HourlyTeamScore       -Dexec.args="--tempLocation=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp --runner=DataflowRunner --input=gs://dataflow-samples/game/5000_gaming_data.csv --project=apache-beam-testing --region=us-central1 --output=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt "       -Pdataflow-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.4
Sep 11, 2021 12:05:33 PM org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
Sep 11, 2021 12:05:34 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 228 files. Enable logging at DEBUG level to see which files will be staged.
Sep 11, 2021 12:05:35 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing implications related to Google Compute Engine usage and other Google Cloud Services.
Sep 11, 2021 12:05:38 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/
Sep 11, 2021 12:05:38 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <235367 bytes, hash 449f6cacdc3aa5d5901514dd4b9f97ef58694693ae1d4342073cbb62ea40f7f5> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-RJ9srNw6pdWQFRTdS5-X71hpRpOuHUNCBzy7YupA9_U.pb
Sep 11, 2021 12:05:40 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Uploading 228 files from PipelineOptions.filesToStage to staging location to prepare for execution.
Sep 11, 2021 12:05:41 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 228 files cached, 0 files newly uploaded in 1 seconds
Sep 11, 2021 12:05:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding TextIO.Read/Read as step s1
Sep 11, 2021 12:05:41 PM org.apache.beam.sdk.io.FileBasedSource getEstimatedSizeBytes
INFO: Filepattern gs://dataflow-samples/game/5000_gaming_data.csv matched 1 files with total size 386400
Sep 11, 2021 12:05:41 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ParseGameEvent as step s2
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding FilterStartTime/ParDo(Anonymous) as step s3
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding FilterEndTime/ParDo(Anonymous) as step s4
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding AddEventTimestamps/AddTimestamps as step s5
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding FixedWindowsTeam/Window.Assign as step s6
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractTeamScore/MapElements/Map as step s7
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey as step s8
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues as step s9
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/ConvertToRow as step s10
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey as step s11
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards as step s12
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles as step s13
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void key/AddKeys/Map as step s14
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign as step s15
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey as step s16
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable as step s17
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop key/Values/Map as step s18
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather bundles as step s19
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair with random key as step s20
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s21
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s22
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s23
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map as step s24
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize as step s25
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s26
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s27
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s28
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s29
Sep 11, 2021 12:05:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s30
Sep 11, 2021 12:05:43 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.34.0-SNAPSHOT
Sep 11, 2021 12:05:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-11_05_05_43-4775761313722959780?project=apache-beam-testing
Sep 11, 2021 12:05:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2021-09-11_05_05_43-4775761313722959780
Sep 11, 2021 12:05:44 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2021-09-11_05_05_43-4775761313722959780
Sep 11, 2021 12:05:49 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:46.659Z: Autoscaling is enabled for job 2021-09-11_05_05_43-4775761313722959780. The number of workers will be between 1 and 1000.
Sep 11, 2021 12:05:49 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:46.763Z: Autoscaling was automatically enabled for job 2021-09-11_05_05_43-4775761313722959780.
Sep 11, 2021 12:05:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:53.179Z: Worker configuration: e2-standard-2 in us-central1-c.
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.085Z: Expanding CoGroupByKey operations into optimizable parts.
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.311Z: Expanding GroupByKey operations into optimizable parts.
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.350Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.587Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.637Z: Fusing consumer ParseGameEvent into TextIO.Read/Read
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.668Z: Fusing consumer FilterStartTime/ParDo(Anonymous) into ParseGameEvent
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.703Z: Fusing consumer FilterEndTime/ParDo(Anonymous) into FilterStartTime/ParDo(Anonymous)
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.751Z: Fusing consumer AddEventTimestamps/AddTimestamps into FilterEndTime/ParDo(Anonymous)
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.790Z: Fusing consumer FixedWindowsTeam/Window.Assign into AddEventTimestamps/AddTimestamps
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.825Z: Fusing consumer ExtractTeamScore/MapElements/Map into FixedWindowsTeam/Window.Assign
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.855Z: Fusing consumer ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey+ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial into ExtractTeamScore/MapElements/Map
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.890Z: Fusing consumer ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Reify into ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey+ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.913Z: Fusing consumer ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Write into ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Reify
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.945Z: Fusing consumer ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues into ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Read
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:54.978Z: Fusing consumer ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract into ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.015Z: Fusing consumer WriteTeamScoreSums/ConvertToRow into ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Extract
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.046Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey into WriteTeamScoreSums/ConvertToRow
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.083Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Reify into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/ApplyShardingKey
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.114Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Write into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Reify
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.147Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/GroupByWindow into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/Read
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.180Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/GroupIntoShards/GroupByWindow
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.202Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void key/AddKeys/Map into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteShardedBundlesToTempFiles/WriteShardsIntoTempFiles
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.234Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void key/AddKeys/Map
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.276Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.314Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.345Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Read
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.368Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.400Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop key/Values/Map into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.427Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather bundles into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop key/Values/Map
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.450Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair with random key into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather bundles
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.483Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair with random key
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.516Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.553Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.591Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.617Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.653Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.686Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.722Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.756Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.789Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.824Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.850Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.874Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:55.929Z: Fusing consumer WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map into WriteTeamScoreSums/WriteToText.WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:56.325Z: Executing operation ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Create
Sep 11, 2021 12:05:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:56.406Z: Starting 1 workers in us-central1-c...
Sep 11, 2021 12:05:59 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:56.852Z: Finished operation ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Create
Sep 11, 2021 12:05:59 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:05:57.065Z: Executing operation TextIO.Read/Read+ParseGameEvent+FilterStartTime/ParDo(Anonymous)+FilterEndTime/ParDo(Anonymous)+AddEventTimestamps/AddTimestamps+FixedWindowsTeam/Window.Assign+ExtractTeamScore/MapElements/Map+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey+ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Reify+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Write
Sep 11, 2021 12:06:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:06:25.126Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
Sep 11, 2021 12:06:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2021-09-11T12:06:30.676Z: Startup of the worker pool in zone us-central1-c failed to bring up any of the desired 1 workers. QUOTA_EXCEEDED: Quota 'CPUS' exceeded.  Limit: 2500.0 in region us-central1.
Sep 11, 2021 12:06:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
SEVERE: 2021-09-11T12:06:30.707Z: Workflow failed.
Sep 11, 2021 12:06:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:06:30.796Z: Finished operation TextIO.Read/Read+ParseGameEvent+FilterStartTime/ParDo(Anonymous)+FilterEndTime/ParDo(Anonymous)+AddEventTimestamps/AddTimestamps+FixedWindowsTeam/Window.Assign+ExtractTeamScore/MapElements/Map+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey+ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues/Partial+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Reify+ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/Write
Sep 11, 2021 12:06:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:06:30.895Z: Cleaning up.
Sep 11, 2021 12:06:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:06:31.022Z: Stopping worker pool...
Sep 11, 2021 12:06:48 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-09-11T12:06:47.596Z: Worker pool stopped.
Sep 11, 2021 12:07:25 PM org.apache.beam.runners.dataflow.DataflowPipelineJob logTerminalState
INFO: Job 2021-09-11_05_05_43-4775761313722959780 failed with status FAILED.
gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt* | grep AzureBilby 
CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt*
CommandException: No URLs matched: gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/java-hourlyteamscore-result-dataflow-runner.txt*
[ERROR] Failed command

> Task :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1h 7m 16s
9 actionable tasks: 8 executed, 1 from cache

Publishing build scan...
https://gradle.com/s/iz4swn6iosdn6

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #1487

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1487/display/redirect?page=changes>

Changes:

[ruwan.lambrichts] Clarify additional_bq_parameters argument

[kawaigin] [BEAM-10708] Support streaming cache in beam_sql magic

[noreply] Fix broken 'differences from pandas' link

[noreply] Added GroupBy row in Aggregation table.

[Luke Cwik] [BEAM-12769] Fix typo in test class name, CLass -> Class

[Etienne Chauchot] [BEAM-5172] Temporary ignore testSplit and testSizes tests waiting for a

[samuelw] [BEAM-12740] Remove matching to filter files when renaming gcs files in

[noreply] [BEAM-3304] Helper functions for triggers (#15430)

[esert] Bump a throttling counter on BigQueryRead retries due to

[noreply] [BEAM-5097] Increment counter for "small words" in go SDK example


------------------------------------------
[...truncated 4.40 MB...]
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:903)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1560)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:994)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1047)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:387)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$800(BatchedStreamingWrite.java:72)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:259)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:903)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1560)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:994)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1047)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:387)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$800(BatchedStreamingWrite.java:72)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:259)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:903)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1560)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:994)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1047)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:387)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$800(BatchedStreamingWrite.java:72)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:259)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:903)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1560)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:994)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1047)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:387)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$800(BatchedStreamingWrite.java:72)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:259)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:903)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1560)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)

Waiting on bqjob_r7962b217db86a5f_0000017bcfb84195_1 ... (0s) Current status: RUNNING
                                                                                     
Waiting on bqjob_r7962b217db86a5f_0000017bcfb84195_1 ... (0s) Current status: DONE   
+-----------------------------+
|            user             |
+-----------------------------+
| user2_AmberKookaburra       |
| user3_BattleshipGreyEchidna |
| user3_AzureWombat           |
| user10_BattleshipGreyBilby  |
| user16_AliceBlueNumbat      |
| Robot-3                     |
| user8_BattleshipGreyEchidna |
| user3_AliceBlueNumbat       |
| user5_RubyCassowary         |
| user18_AmethystKookaburra   |
+-----------------------------+
Verified AliceBlue
RUNNING_JOB=`gcloud dataflow jobs list | grep leaderboard-validation-1631276661662-594 | grep Running | cut -d' ' -f1`
if [ ! -z "${RUNNING_JOB}" ] 
  then 
    gcloud dataflow jobs cancel ${RUNNING_JOB}
  else 
    echo "Job 'leaderboard-validation-1631276661662-594' is not running."
fi 

Cancelled job [2021-09-10_05_25_10-16678837790518394442]

**************************************
* SUCCESS: LeaderBoard successfully run on DataflowRunner. with Streaming Engine
**************************************

[SUCCESS]

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1h 39m 11s
9 actionable tasks: 8 executed, 1 from cache

Publishing build scan...
https://gradle.com/s/5kidwiw2i2qig

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org