You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2018/11/17 00:48:08 UTC
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1062
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1062/display/redirect?page=changes>
Changes:
[valentyn] Change Dataflow container prefix.
------------------------------------------
[...truncated 263.26 KB...]
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 17, 2018 12:42:21 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-JO3tp5zh08qhRSRIYeaasw.jar
Nov 17, 2018 12:42:23 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-JO3tp5zh08qhRSRIYeaasw.jar
Nov 17, 2018 12:42:26 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 17, 2018 12:42:27 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88173 bytes, hash 00eU-nqMeGyRKmEGydrCEA> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-00eU-nqMeGyRKmEGydrCEA.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 17, 2018 12:42:28 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-16_16_42_27-9790361880214550457
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-16_16_42_27-9790361880214550457?project=apache-beam-testing
Nov 17, 2018 12:42:28 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-16_16_42_27-9790361880214550457
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-17T00:42:27.817Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:30.972Z: Checking permissions granted to controller Service Account.
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:35.235Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:35.698Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:35.808Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:35.837Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.036Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.088Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.124Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.190Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.248Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.342Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.377Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.425Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.488Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.532Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.585Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.627Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.672Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.724Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.771Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.819Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.888Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.938Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:36.994Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.048Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.097Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.147Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.203Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.250Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.301Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.354Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.400Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.436Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.484Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.539Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.587Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.635Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.678Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:37.730Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:38.261Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:38.315Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:38.363Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:38.363Z: Starting 4 workers in us-central1-b...
Nov 17, 2018 12:42:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:38.687Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 17, 2018 12:42:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:42:50.758Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 17, 2018 12:43:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:43:22.411Z: Workers have started successfully.
Nov 17, 2018 12:43:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:43:31.059Z: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s).
Nov 17, 2018 12:43:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:43:31.107Z: Resized worker pool to 2, though goal was 4. This could be a quota issue.
Nov 17, 2018 12:43:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:43:36.788Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 17, 2018 12:43:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:43:36.839Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 17, 2018 12:43:43 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:43:42.734Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 17, 2018 12:43:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:43:53.678Z: Workers have started successfully.
Nov 17, 2018 12:44:46 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:44:45.531Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 17, 2018 12:44:46 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:44:45.637Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 17, 2018 12:44:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:44:47.343Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 17, 2018 12:44:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:44:47.443Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 17, 2018 12:44:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:44:48.949Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 17, 2018 12:44:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:44:49.047Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 17, 2018 12:44:53 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:44:52.112Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 17, 2018 12:44:53 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:44:52.336Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 17, 2018 12:45:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:45:17.864Z: Cleaning up.
Nov 17, 2018 12:45:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T00:45:17.966Z: Stopping worker pool...
Nov 17, 2018 12:46:28 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-17T00:46:28.879Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-17T00:46:28.914Z RUNNING Query7
2018-11-17T00:46:30.276Z new perf {"snapshots":null,"runtimeSec":101.063,"numEvents":10000000,"eventsPerSec":98948.1808376953,"eventBytesPerSec":1.2700641154527374E7,"numResults":100,"resultsPerSec":0.989481808376953,"resultBytesPerSec":99.96734710032356,"startupDelaySec":-172.582,"processingDelaySec":77.831,"shutdownDelaySec":71.519,"timeDilation":9.730564103578955,"errors":null,"jobId":null}
2018-11-17T00:46:45.322Z RUNNING Query7
2018-11-17T00:46:46.400Z no activity
2018-11-17T00:47:01.442Z RUNNING Query7
2018-11-17T00:47:02.837Z no activity
2018-11-17T00:47:17.874Z RUNNING Query7
2018-11-17T00:47:18.984Z no activity
2018-11-17T00:47:34.023Z RUNNING Query7
2018-11-17T00:47:35.094Z no activity
2018-11-17T00:47:50.133Z RUNNING Query7
2018-11-17T00:47:51.197Z no activity
2018-11-17T00:48:06.259Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 9,5,main]) completed. Took 6 mins 7.476 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 55s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/73tsiydiydlcc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal :
beam_PostCommit_Java_Nexmark_Dataflow #1092
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1092/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1091
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1091/display/redirect?page=changes>
Changes:
[echauchot] [BEAM-5969] Fix typo in nexmark web page
[mxm] [website] Add 'builds' mailing list to the contact page
------------------------------------------
[...truncated 381.91 KB...]
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys as step s11
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) as step s12
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections as step s13
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView as step s14
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) as step s15
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/AsBid as step s16
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles as step s17
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Debug as step s18
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Stamp as step s19
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.Format as step s20
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.DevNull as step s21
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <58906 bytes, hash 4k0HKoRdW0mkWLHwM962Ng> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-4k0HKoRdW0mkWLHwM962Ng.pb
Nov 20, 2018 10:04:04 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88194 bytes, hash A5XVO8-mvoxNTyTEHhSaxw> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-A5XVO8-mvoxNTyTEHhSaxw.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Dataflow SDK version: 2.9.0-SNAPSHOT
Submitted job: 2018-11-20_02_04_05-9943991727944894115
Nov 20, 2018 10:04:06 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-20_02_04_05-9943991727944894115?project=apache-beam-testing
Nov 20, 2018 10:04:06 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-20_02_04_05-9943991727944894115
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-20T10:04:05.084Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:08.522Z: Checking permissions granted to controller Service Account.
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:13.439Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.154Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.238Z: Expanding GroupByKey operations into optimizable parts.
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.288Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.472Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.515Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#25398a999f74762b
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.557Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.608Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.656Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.695Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.730Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.768Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.814Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.861Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.896Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.937Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:14.982Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.030Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.083Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.126Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.165Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.201Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.239Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.288Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.339Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.383Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.417Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.468Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.511Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.543Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.585Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.632Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.666Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.715Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.758Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.800Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.849Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:15.887Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:16.407Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:16.446Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:16.484Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:16.517Z: Starting 4 workers in us-central1-b...
Nov 20, 2018 10:04:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:16.779Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 20, 2018 10:04:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:04:35.498Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 20, 2018 10:05:02 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:05:02.325Z: Workers have started successfully.
Nov 20, 2018 10:05:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:05:04.411Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 20, 2018 10:05:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:05:04.454Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 20, 2018 10:05:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:05:05.226Z: Workers have started successfully.
Nov 20, 2018 10:05:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:05:21.668Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 20, 2018 10:06:19 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:06:17.925Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 20, 2018 10:06:19 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:06:18.021Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 20, 2018 10:06:19 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:06:19.681Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 20, 2018 10:06:19 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:06:19.777Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 20, 2018 10:06:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:06:21.212Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 20, 2018 10:06:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:06:21.308Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 20, 2018 10:06:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:06:23.718Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 20, 2018 10:06:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:06:23.951Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 20, 2018 10:06:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:06:51.715Z: Cleaning up.
Nov 20, 2018 10:06:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T10:06:51.853Z: Stopping worker pool...
Nov 20, 2018 10:08:06 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-20T10:08:06.589Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-20T10:08:06.624Z RUNNING Query7
2018-11-20T10:08:07.781Z new perf {"snapshots":null,"runtimeSec":95.68,"numEvents":10000000,"eventsPerSec":104515.05016722408,"eventBytesPerSec":1.341518496028428E7,"numResults":100,"resultsPerSec":1.0451505016722407,"resultBytesPerSec":105.59155518394648,"startupDelaySec":-171.076,"processingDelaySec":70.044,"shutdownDelaySec":75.396,"timeDilation":10.278010033444815,"errors":null,"jobId":null}
2018-11-20T10:08:22.826Z RUNNING Query7
2018-11-20T10:08:24.042Z no activity
2018-11-20T10:08:39.080Z RUNNING Query7
2018-11-20T10:08:40.100Z no activity
2018-11-20T10:08:55.147Z RUNNING Query7
2018-11-20T10:08:56.208Z no activity
2018-11-20T10:09:11.264Z RUNNING Query7
2018-11-20T10:09:12.311Z no activity
2018-11-20T10:09:27.351Z RUNNING Query7
2018-11-20T10:09:28.445Z no activity
2018-11-20T10:09:43.486Z RUNNING Query7
2018-11-20T10:09:44.495Z no activity
2018-11-20T10:09:59.544Z RUNNING Query7
2018-11-20T10:09:59.705Z no activity
2018-11-20T10:10:14.706Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 10,5,main]) completed. Took 6 mins 39.386 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 10m 55s
76 actionable tasks: 72 executed, 4 from cache
Publishing build scan...
https://gradle.com/s/6l2dekckqdki6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1090
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1090/display/redirect?page=changes>
Changes:
[relax] CoGroup transforms for schemas.
------------------------------------------
[...truncated 233.61 KB...]
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-SJ0iKljRwJqRaVO78HFu2A.jar
Nov 20, 2018 8:50:20 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-SJ0iKljRwJqRaVO78HFu2A.jar
Nov 20, 2018 8:50:20 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-SJ0iKljRwJqRaVO78HFu2A.jar
Nov 20, 2018 8:50:20 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-SJ0iKljRwJqRaVO78HFu2A.jar
Nov 20, 2018 8:50:20 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-SJ0iKljRwJqRaVO78HFu2A.jar
Nov 20, 2018 8:50:20 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-SJ0iKljRwJqRaVO78HFu2A.jar
Nov 20, 2018 8:50:20 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-SJ0iKljRwJqRaVO78HFu2A.jar
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 117 files cached, 29 files newly uploaded
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 20, 2018 8:50:24 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 20, 2018 8:50:25 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 20, 2018 8:50:25 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 20, 2018 8:50:25 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 20, 2018 8:50:25 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88179 bytes, hash a_H6UnUYycCHXUZUuqLt5Q> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-a_H6UnUYycCHXUZUuqLt5Q.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 20, 2018 8:50:29 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-20_00_50_25-6387333694799286878
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-20_00_50_25-6387333694799286878?project=apache-beam-testing
Nov 20, 2018 8:50:29 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-20_00_50_25-6387333694799286878
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-20T08:50:25.592Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:32.213Z: Checking permissions granted to controller Service Account.
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:38.389Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:38.872Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:38.945Z: Expanding GroupByKey operations into optimizable parts.
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:38.978Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.171Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.207Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.243Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.277Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.313Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.350Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.386Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.424Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.458Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.483Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.515Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.548Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.585Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.619Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.656Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.692Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.729Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.764Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.801Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.838Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.876Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.916Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.949Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:39.993Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.027Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.063Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.101Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.132Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.169Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.204Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.242Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.280Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.312Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.348Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.744Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.781Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.822Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:40.829Z: Starting 4 workers in us-central1-b...
Nov 20, 2018 8:50:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:41.083Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 20, 2018 8:50:53 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:50:53.500Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 20, 2018 8:51:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:51:23.338Z: Workers have started successfully.
Nov 20, 2018 8:51:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:51:27.376Z: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s).
Nov 20, 2018 8:51:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:51:27.418Z: Resized worker pool to 2, though goal was 4. This could be a quota issue.
Nov 20, 2018 8:51:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:51:30.461Z: Workers have started successfully.
Nov 20, 2018 8:51:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:51:39.489Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 20, 2018 8:51:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:51:39.522Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 20, 2018 8:51:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:51:45.228Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 20, 2018 8:52:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:52:44.749Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 20, 2018 8:52:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:52:44.868Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 20, 2018 8:52:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:52:46.726Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 20, 2018 8:52:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:52:46.843Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 20, 2018 8:52:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:52:49.417Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 20, 2018 8:52:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:52:49.539Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 20, 2018 8:52:53 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:52:53.090Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 20, 2018 8:52:53 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:52:53.340Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 20, 2018 8:53:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:53:18.672Z: Cleaning up.
Nov 20, 2018 8:53:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T08:53:18.778Z: Stopping worker pool...
Nov 20, 2018 8:54:30 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-20T08:54:30.056Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-20T08:54:30.094Z RUNNING Query7
2018-11-20T08:54:31.164Z new perf {"snapshots":null,"runtimeSec":101.228,"numEvents":10000000,"eventsPerSec":98786.89690599439,"eventBytesPerSec":1.2679939315209232E7,"numResults":100,"resultsPerSec":0.9878689690599439,"resultBytesPerSec":99.80440194412614,"startupDelaySec":-172.957,"processingDelaySec":77.905,"shutdownDelaySec":71.729,"timeDilation":9.71470344173549,"errors":null,"jobId":null}
2018-11-20T08:54:46.204Z RUNNING Query7
2018-11-20T08:54:47.255Z no activity
2018-11-20T08:55:02.287Z RUNNING Query7
2018-11-20T08:55:03.369Z no activity
2018-11-20T08:55:18.407Z RUNNING Query7
2018-11-20T08:55:19.524Z no activity
2018-11-20T08:55:34.561Z RUNNING Query7
2018-11-20T08:55:35.599Z no activity
2018-11-20T08:55:50.649Z RUNNING Query7
2018-11-20T08:55:51.681Z no activity
2018-11-20T08:56:06.736Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 5 mins 49.985 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 35s
76 actionable tasks: 72 executed, 4 from cache
Publishing build scan...
https://gradle.com/s/zqqk2rkc4okkw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1089
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1089/display/redirect>
------------------------------------------
[...truncated 257.91 KB...]
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-vHRCqotCn6lvBiqXwBTqKQ.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:512)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 20, 2018 6:03:54 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-vHRCqotCn6lvBiqXwBTqKQ.jar
Nov 20, 2018 6:03:58 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-vHRCqotCn6lvBiqXwBTqKQ.jar
Nov 20, 2018 6:04:01 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 117 files cached, 29 files newly uploaded
Nov 20, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 20, 2018 6:04:02 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88182 bytes, hash w5soxkiSkC8m5e06Xa6Inw> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-w5soxkiSkC8m5e06Xa6Inw.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 20, 2018 6:04:03 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-19_22_04_02-14667256735523976504
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_22_04_02-14667256735523976504?project=apache-beam-testing
Nov 20, 2018 6:04:03 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_22_04_02-14667256735523976504
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-20T06:04:02.741Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:05.777Z: Checking permissions granted to controller Service Account.
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:10.930Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:11.444Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:11.534Z: Expanding GroupByKey operations into optimizable parts.
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:11.573Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:11.773Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:11.822Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#a70ce18896339064
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:11.870Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:11.917Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:11.965Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.010Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.056Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.104Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.146Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.181Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.218Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.257Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.303Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.349Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.397Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.435Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.483Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.523Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.568Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.616Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.651Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.696Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.742Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.788Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.835Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.875Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.922Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:12.969Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:13.018Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:13.054Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:13.100Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:13.149Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:13.186Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:13.219Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:13.679Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:13.726Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:13.775Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:13.786Z: Starting 4 workers in us-central1-b...
Nov 20, 2018 6:04:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:14.095Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 20, 2018 6:04:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:33.168Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 20, 2018 6:04:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:56.598Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 20, 2018 6:04:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:04:56.634Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 20, 2018 6:05:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:05:02.650Z: Workers have started successfully.
Nov 20, 2018 6:05:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:05:02.694Z: Workers have started successfully.
Nov 20, 2018 6:05:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:05:14.624Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 20, 2018 6:06:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:06:19.742Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 20, 2018 6:06:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:06:19.846Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 20, 2018 6:06:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:06:20.624Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 20, 2018 6:06:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:06:20.728Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 20, 2018 6:06:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:06:22.462Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 20, 2018 6:06:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:06:22.564Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 20, 2018 6:06:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:06:25.841Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 20, 2018 6:06:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:06:26.031Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 20, 2018 6:06:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:06:54.183Z: Cleaning up.
Nov 20, 2018 6:06:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T06:06:54.334Z: Stopping worker pool...
Nov 20, 2018 6:08:03 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-20T06:08:03.694Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-20T06:08:03.728Z RUNNING Query7
2018-11-20T06:08:04.798Z new perf {"snapshots":null,"runtimeSec":96.918,"numEvents":10000000,"eventsPerSec":103180.00784168059,"eventBytesPerSec":1.3243823613776594E7,"numResults":100,"resultsPerSec":1.031800078416806,"resultBytesPerSec":104.2427619224499,"startupDelaySec":-167.159,"processingDelaySec":72.389,"shutdownDelaySec":70.241,"timeDilation":10.14672197115087,"errors":null,"jobId":null}
2018-11-20T06:08:19.844Z RUNNING Query7
2018-11-20T06:08:20.815Z no activity
2018-11-20T06:08:35.851Z RUNNING Query7
2018-11-20T06:08:36.929Z no activity
2018-11-20T06:08:51.969Z RUNNING Query7
2018-11-20T06:08:53.016Z no activity
2018-11-20T06:09:08.055Z RUNNING Query7
2018-11-20T06:09:09.152Z no activity
2018-11-20T06:09:24.189Z RUNNING Query7
2018-11-20T06:09:25.222Z no activity
2018-11-20T06:09:40.273Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 4,5,main]) completed. Took 6 mins 0.225 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 33s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/xqoamggzskfhs
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1088
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1088/display/redirect?page=changes>
Changes:
[cushon] [BEAM-6096] illegal signature attribute when compiling with JDK 11
------------------------------------------
[...truncated 269.74 KB...]
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-JXk4_AkNyZ8UWLy68oYasg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-JXk4_AkNyZ8UWLy68oYasg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:432)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:287)
at org.apache.beam.runners.dataflow.util.PackageUtil.$closeResource(PackageUtil.java:260)
at org.apache.beam.runners.dataflow.util.PackageUtil.tryStagePackage(PackageUtil.java:260)
at org.apache.beam.runners.dataflow.util.PackageUtil.tryStagePackageWithRetry(PackageUtil.java:203)
at org.apache.beam.runners.dataflow.util.PackageUtil.stagePackageSynchronously(PackageUtil.java:187)
at org.apache.beam.runners.dataflow.util.PackageUtil.lambda$stagePackage$1(PackageUtil.java:171)
at org.apache.beam.sdk.util.MoreFutures.lambda$supplyAsync$0(MoreFutures.java:104)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-JXk4_AkNyZ8UWLy68oYasg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-JXk4_AkNyZ8UWLy68oYasg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:512)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 20, 2018 5:51:47 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-JXk4_AkNyZ8UWLy68oYasg.jar
Nov 20, 2018 5:51:49 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-JXk4_AkNyZ8UWLy68oYasg.jar
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 117 files cached, 28 files newly uploaded
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3.ReadBounded as step s1
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3.Monitor as step s2
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3.Snoop as step s3
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Window.Into()/Window.Assign as step s4
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/justNewAuctions/IsNewAuction/ParDo(Anonymous) as step s5
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/justNewAuctions/AsAuction as step s6
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.InCategory/ParDo(Anonymous) as step s7
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/AuctionBySeller as step s8
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/justNewPersons/IsNewPerson/ParDo(Anonymous) as step s9
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/justNewPersons/AsPerson as step s10
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.InState/ParDo(Anonymous) as step s11
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/PersonById as step s12
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/CoGroupByKey/MakeUnionTable0 as step s13
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/CoGroupByKey/MakeUnionTable1 as step s14
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/CoGroupByKey/Flatten as step s15
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/CoGroupByKey/GBK as step s16
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/CoGroupByKey/ConstructCoGbkResultFn as step s17
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/ReifyWindows as step s18
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys as step s19
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.Join/ParDo(BatchStateful) as step s20
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.Project as step s21
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3.Debug as step s22
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3.Stamp as step s23
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3.Format as step s24
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3.DevNull as step s25
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 20, 2018 5:51:53 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <65289 bytes, hash E5tK16nmdPiAqXp-J5KmiA> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-E5tK16nmdPiAqXp-J5KmiA.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 20, 2018 5:51:55 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-19_21_51_54-3518111521880910996
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_21_51_54-3518111521880910996?project=apache-beam-testing
Nov 20, 2018 5:51:55 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_21_51_54-3518111521880910996
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-20T05:51:54.466Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:51:57.004Z: Checking permissions granted to controller Service Account.
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:03.349Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:03.997Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.096Z: Expanding GroupByKey operations into optimizable parts.
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.118Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.241Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.281Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#9a88db53ad38536b
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.316Z: Fusing unzipped copy of Query3/Query3/CoGroupByKey/GBK/Reify, through flatten Query3/Query3/CoGroupByKey/Flatten, into producer Query3/Query3/CoGroupByKey/MakeUnionTable0
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.351Z: Fusing consumer Query3/Query3.Stamp into Query3/Query3.Debug
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.388Z: Fusing consumer Query3.Format into Query3/Query3.Stamp
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.423Z: Unzipping flatten s15-u18 for input s16-reify-value0-c16
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.451Z: Fusing unzipped copy of Query3/Query3/CoGroupByKey/GBK/Write, through flatten s15-u18, into producer Query3/Query3/CoGroupByKey/GBK/Reify
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.484Z: Fusing consumer Query3.DevNull into Query3.Format
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.511Z: Fusing consumer Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Write into Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/ReifyWindows
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.548Z: Fusing consumer Query3/Query3/Query3.Project into Query3/Query3/Query3.Join/ParDo(BatchStateful)
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.586Z: Fusing consumer Query3/Query3.Debug into Query3/Query3/Query3.Project
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.618Z: Fusing consumer Query3/Query3/Query3.Join/ParDo(BatchStateful) into Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Read
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.656Z: Fusing consumer Query3/Query3/CoGroupByKey/GBK/GroupByWindow into Query3/Query3/CoGroupByKey/GBK/Read
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.693Z: Fusing consumer Query3/Query3/CoGroupByKey/ConstructCoGbkResultFn into Query3/Query3/CoGroupByKey/GBK/GroupByWindow
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.730Z: Fusing consumer Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/ReifyWindows into Query3/Query3/CoGroupByKey/ConstructCoGbkResultFn
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.769Z: Fusing consumer Query3/Query3/CoGroupByKey/GBK/Reify into Query3/Query3/CoGroupByKey/MakeUnionTable1
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.822Z: Fusing consumer Query3/Query3/CoGroupByKey/GBK/Write into Query3/Query3/CoGroupByKey/GBK/Reify
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.850Z: Fusing consumer Query3/Query3/justNewAuctions/IsNewAuction/ParDo(Anonymous) into Query3/Query3/Window.Into()/Window.Assign
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.892Z: Fusing consumer Query3/Query3/justNewPersons/IsNewPerson/ParDo(Anonymous) into Query3/Query3/Window.Into()/Window.Assign
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.926Z: Fusing consumer Query3/Query3/Query3.InCategory/ParDo(Anonymous) into Query3/Query3/justNewAuctions/AsAuction
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.961Z: Fusing consumer Query3/Query3.Snoop into Query3/Query3.Monitor
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:04.996Z: Fusing consumer Query3/Query3/PersonById into Query3/Query3/Query3.InState/ParDo(Anonymous)
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.032Z: Fusing consumer Query3/Query3/CoGroupByKey/MakeUnionTable1 into Query3/Query3/PersonById
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.072Z: Fusing consumer Query3/Query3/justNewAuctions/AsAuction into Query3/Query3/justNewAuctions/IsNewAuction/ParDo(Anonymous)
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.098Z: Fusing consumer Query3/Query3/Query3.InState/ParDo(Anonymous) into Query3/Query3/justNewPersons/AsPerson
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.134Z: Fusing consumer Query3/Query3/justNewPersons/AsPerson into Query3/Query3/justNewPersons/IsNewPerson/ParDo(Anonymous)
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.171Z: Fusing consumer Query3/Query3/AuctionBySeller into Query3/Query3/Query3.InCategory/ParDo(Anonymous)
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.208Z: Fusing consumer Query3/Query3.Monitor into Query3.ReadBounded
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.243Z: Fusing consumer Query3/Query3/Window.Into()/Window.Assign into Query3/Query3.Snoop
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.281Z: Fusing consumer Query3/Query3/CoGroupByKey/MakeUnionTable0 into Query3/Query3/AuctionBySeller
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.721Z: Executing operation Query3/Query3/CoGroupByKey/GBK/Create
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.766Z: Executing operation Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Create
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:05.818Z: Starting 4 workers in us-central1-b...
Nov 20, 2018 5:52:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:06.004Z: Executing operation Query3.ReadBounded+Query3/Query3.Monitor+Query3/Query3.Snoop+Query3/Query3/Window.Into()/Window.Assign+Query3/Query3/justNewAuctions/IsNewAuction/ParDo(Anonymous)+Query3/Query3/justNewPersons/IsNewPerson/ParDo(Anonymous)+Query3/Query3/justNewAuctions/AsAuction+Query3/Query3/Query3.InCategory/ParDo(Anonymous)+Query3/Query3/justNewPersons/AsPerson+Query3/Query3/Query3.InState/ParDo(Anonymous)+Query3/Query3/PersonById+Query3/Query3/CoGroupByKey/MakeUnionTable1+Query3/Query3/CoGroupByKey/GBK/Reify+Query3/Query3/AuctionBySeller+Query3/Query3/CoGroupByKey/MakeUnionTable0+Query3/Query3/CoGroupByKey/GBK/Reify+Query3/Query3/CoGroupByKey/GBK/Session/Flatten+Query3/Query3/CoGroupByKey/GBK/Write
Nov 20, 2018 5:52:23 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:22.985Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 20, 2018 5:52:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:46.717Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 20, 2018 5:52:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:46.764Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 20, 2018 5:52:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:49.530Z: Workers have started successfully.
Nov 20, 2018 5:52:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:49.568Z: Workers have started successfully.
Nov 20, 2018 5:52:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:58.722Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 20, 2018 5:52:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:52:58.762Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 20, 2018 5:53:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:53:05.127Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 20, 2018 5:53:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:53:44.084Z: Executing operation Query3/Query3/CoGroupByKey/GBK/Close
Nov 20, 2018 5:53:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:53:44.183Z: Executing operation Query3/Query3/CoGroupByKey/GBK/Read+Query3/Query3/CoGroupByKey/GBK/GroupByWindow+Query3/Query3/CoGroupByKey/ConstructCoGbkResultFn+Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/ReifyWindows+Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Write
Nov 20, 2018 5:53:50 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:53:50.103Z: Executing operation Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Close
Nov 20, 2018 5:53:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:53:50.200Z: Executing operation Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Read+Query3/Query3/Query3.Join/ParDo(BatchStateful)+Query3/Query3/Query3.Project+Query3/Query3.Debug+Query3/Query3.Stamp+Query3.Format+Query3.DevNull
Nov 20, 2018 5:53:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:53:53.892Z: Cleaning up.
Nov 20, 2018 5:53:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:53:53.997Z: Stopping worker pool...
Nov 20, 2018 5:55:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:55:53.597Z: Autoscaling: Resized worker pool from 4 to 0.
Nov 20, 2018 5:55:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T05:55:53.647Z: Worker pool stopped.
Nov 20, 2018 5:55:55 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-20T05:55:55.387Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-20T05:55:55.415Z RUNNING Query3
2018-11-20T05:55:56.633Z new perf {"snapshots":null,"runtimeSec":49.931,"numEvents":10000000,"eventsPerSec":200276.38140634075,"eventBytesPerSec":2.570677328713625E7,"numResults":59507,"resultsPerSec":1191.784662834712,"resultBytesPerSec":38085.918567623325,"startupDelaySec":-171.659,"processingDelaySec":47.128,"shutdownDelaySec":121.728,"timeDilation":-1.0,"errors":null,"jobId":null}
2018-11-20T05:56:11.690Z DONE Query3
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 4 mins 46.303 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 8m 18s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/ko5qqku753iqa
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1087
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1087/display/redirect>
------------------------------------------
[...truncated 240.14 KB...]
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-3MGJVaahA-SBw4a7T_wT-w.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:512)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 20, 2018 12:03:57 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-3MGJVaahA-SBw4a7T_wT-w.jar
Nov 20, 2018 12:04:00 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 117 files cached, 29 files newly uploaded
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 20, 2018 12:04:01 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88175 bytes, hash c0xVojXhSoNAj9385lvUdw> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-c0xVojXhSoNAj9385lvUdw.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 20, 2018 12:04:03 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-19_16_04_02-11739024428374523017
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_16_04_02-11739024428374523017?project=apache-beam-testing
Nov 20, 2018 12:04:03 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_16_04_02-11739024428374523017
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-20T00:04:02.444Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:05.325Z: Checking permissions granted to controller Service Account.
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:12.471Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.114Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.213Z: Expanding GroupByKey operations into optimizable parts.
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.254Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.473Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.524Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#7f4de336599327c8
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.558Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.598Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.651Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.700Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.738Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.790Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.826Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.860Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.891Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.932Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.970Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:13.999Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.041Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.092Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.124Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.157Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.196Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.238Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.278Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.305Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.341Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.382Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.418Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.458Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.491Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.533Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.571Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.609Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.632Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.663Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.700Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:14.734Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:15.151Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:15.193Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:15.228Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:15.229Z: Starting 4 workers in us-central1-b...
Nov 20, 2018 12:04:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:15.501Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 20, 2018 12:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:25.611Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 20, 2018 12:04:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:48.487Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 20, 2018 12:04:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:48.529Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 20, 2018 12:04:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:51.125Z: Workers have started successfully.
Nov 20, 2018 12:04:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:04:51.223Z: Workers have started successfully.
Nov 20, 2018 12:05:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:05:05.891Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 20, 2018 12:05:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:05:05.921Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 20, 2018 12:05:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:05:12.083Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 20, 2018 12:06:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:06:11.970Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 20, 2018 12:06:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:06:12.094Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 20, 2018 12:06:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:06:13.613Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 20, 2018 12:06:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:06:13.715Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 20, 2018 12:06:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:06:15.052Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 20, 2018 12:06:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:06:15.156Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 20, 2018 12:06:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:06:18.774Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 20, 2018 12:06:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:06:18.940Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 20, 2018 12:06:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:06:48.120Z: Cleaning up.
Nov 20, 2018 12:06:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-20T00:06:48.325Z: Stopping worker pool...
Nov 20, 2018 12:08:03 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-20T00:08:03.376Z Waiting for main pipeline to 'finish'WARNING: No terminal state was returned. State value RUNNING
2018-11-20T00:08:03.430Z RUNNING Query7
2018-11-20T00:08:04.882Z new perf {"snapshots":null,"runtimeSec":103.491,"numEvents":10000000,"eventsPerSec":96626.75981486312,"eventBytesPerSec":1.2402671700920854E7,"numResults":100,"resultsPerSec":0.9662675981486313,"resultBytesPerSec":97.62201544095622,"startupDelaySec":-179.111,"processingDelaySec":76.143,"shutdownDelaySec":75.62,"timeDilation":9.50227556019364,"errors":null,"jobId":null}
2018-11-20T00:08:19.943Z RUNNING Query7
2018-11-20T00:08:20.979Z no activity
2018-11-20T00:08:36.022Z RUNNING Query7
2018-11-20T00:08:37.123Z no activity
2018-11-20T00:08:52.187Z RUNNING Query7
2018-11-20T00:08:53.397Z no activity
2018-11-20T00:09:08.447Z RUNNING Query7
2018-11-20T00:09:09.717Z no activity
2018-11-20T00:09:24.792Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 4,5,main]) completed. Took 5 mins 39.71 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 18s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/hhnnwmmdjm4zy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1086
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1086/display/redirect?page=changes>
Changes:
[amaliujia] [BEAM-6083] use same parsing functions to parse time
------------------------------------------
[...truncated 255.47 KB...]
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-vdAjqrH4BN5zUY_FaQ8Lpw.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:512)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 19, 2018 11:22:31 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-vdAjqrH4BN5zUY_FaQ8Lpw.jar
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 117 files cached, 29 files newly uploaded
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 11:22:34 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88172 bytes, hash g-nfkRBQXM_w88uHQO_-DQ> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-g-nfkRBQXM_w88uHQO_-DQ.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Submitted job: 2018-11-19_15_22_35-247060295108997909
Nov 19, 2018 11:22:36 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_15_22_35-247060295108997909?project=apache-beam-testing
Nov 19, 2018 11:22:36 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_15_22_35-247060295108997909
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T23:22:35.334Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:38.443Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.004Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.467Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.551Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.596Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.747Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.779Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#84744397f8dc9244
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.806Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.835Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.877Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.908Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.941Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:46.979Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.008Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.043Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.076Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.104Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.140Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.177Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.214Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.251Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.288Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.328Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.361Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.398Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.426Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.457Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.494Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.529Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.565Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.599Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.621Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.652Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.678Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.713Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.740Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.771Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.807Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:47.844Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:48.223Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:48.245Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:48.281Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:48.297Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 11:22:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:48.525Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 19, 2018 11:22:58 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:22:58.758Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 11:23:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:23:24.845Z: Workers have started successfully.
Nov 19, 2018 11:23:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:23:27.446Z: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s).
Nov 19, 2018 11:23:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:23:27.484Z: Resized worker pool to 2, though goal was 4. This could be a quota issue.
Nov 19, 2018 11:23:41 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:23:39.038Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 19, 2018 11:23:41 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:23:39.088Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 19, 2018 11:23:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:23:45.144Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 11:23:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:23:55.120Z: Workers have started successfully.
Nov 19, 2018 11:24:48 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:24:47.849Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 19, 2018 11:24:48 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:24:47.943Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 19, 2018 11:24:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:24:48.772Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 19, 2018 11:24:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:24:48.864Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 19, 2018 11:24:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:24:50.501Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 19, 2018 11:24:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:24:50.585Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 19, 2018 11:24:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:24:54.853Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 19, 2018 11:24:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:24:55.036Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 19, 2018 11:25:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:25:25.029Z: Cleaning up.
Nov 19, 2018 11:25:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T23:25:25.126Z: Stopping worker pool...
Nov 19, 2018 11:26:36 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
WARNING: No terminal state was returned. State value RUNNING
2018-11-19T23:26:36.314Z Waiting for main pipeline to 'finish'
2018-11-19T23:26:36.372Z RUNNING Query7
2018-11-19T23:26:37.528Z new perf {"snapshots":null,"runtimeSec":105.712,"numEvents":10000000,"eventsPerSec":94596.63992734978,"eventBytesPerSec":1.2142092638489481E7,"numResults":100,"resultsPerSec":0.9459663992734978,"resultBytesPerSec":95.57098531860149,"startupDelaySec":-177.277,"processingDelaySec":77.988,"shutdownDelaySec":71.565,"timeDilation":9.302633570455576,"errors":null,"jobId":null}
2018-11-19T23:26:52.582Z RUNNING Query7
2018-11-19T23:26:53.767Z no activity
2018-11-19T23:27:08.806Z RUNNING Query7
2018-11-19T23:27:10.015Z no activity
2018-11-19T23:27:25.071Z RUNNING Query7
2018-11-19T23:27:26.405Z no activity
2018-11-19T23:27:41.450Z RUNNING Query7
2018-11-19T23:27:42.568Z no activity
2018-11-19T23:27:57.627Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 10,5,main]) completed. Took 5 mins 48.133 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 37s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/347rvz4fodqxo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1085
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1085/display/redirect?page=changes>
Changes:
[batbat] Upgraded apiary client version to 1.27.0.
------------------------------------------
[...truncated 266.06 KB...]
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-VzKLYIwPSK5cylYEBQRs5Q.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-VzKLYIwPSK5cylYEBQRs5Q.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:512)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 19, 2018 9:15:38 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-VzKLYIwPSK5cylYEBQRs5Q.jar
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 97 files cached, 49 files newly uploaded
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 9:15:42 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88191 bytes, hash RTS9wig63A0CD56uVO1N3w> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-RTS9wig63A0CD56uVO1N3w.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Submitted job: 2018-11-19_13_15_43-773274572652502452
Nov 19, 2018 9:15:43 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_13_15_43-773274572652502452?project=apache-beam-testing
Nov 19, 2018 9:15:43 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_13_15_43-773274572652502452
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T21:15:43.184Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:45.875Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:52.504Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.042Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.149Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.185Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.382Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.426Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#37bc5e2e3faec20a
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.471Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.528Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.582Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.629Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.691Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.740Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.797Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.849Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.899Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:53.950Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.003Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.055Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.112Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.161Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.207Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.266Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.313Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.358Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.406Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.482Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.530Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.585Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.681Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.739Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.780Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.834Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.877Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.919Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:54.969Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:55.012Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:55.059Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:55.162Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:55.727Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:55.771Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:55.823Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:55.835Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 9:16:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:15:56.125Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 19, 2018 9:16:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:16:08.234Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 9:16:44 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:16:42.200Z: Workers have started successfully.
Nov 19, 2018 9:16:49 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:16:49.206Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 9:17:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:17:12.375Z: Workers have started successfully.
Nov 19, 2018 9:18:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:17:59.383Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 19, 2018 9:18:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:17:59.488Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 19, 2018 9:18:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:18:00.203Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 19, 2018 9:18:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:18:00.291Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 19, 2018 9:18:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:18:02.727Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 19, 2018 9:18:03 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:18:02.808Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 19, 2018 9:18:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:18:05.805Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 19, 2018 9:18:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:18:06.001Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 19, 2018 9:18:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:18:33.122Z: Cleaning up.
Nov 19, 2018 9:18:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T21:18:33.233Z: Stopping worker pool...
Nov 19, 2018 9:19:44 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-19T21:19:44.023Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-19T21:19:44.070Z RUNNING Query7
2018-11-19T21:19:45.343Z new perf {"snapshots":null,"runtimeSec":96.596,"numEvents":10000000,"eventsPerSec":103523.95544328957,"eventBytesPerSec":1.3287971520559857E7,"numResults":100,"resultsPerSec":1.0352395544328958,"resultBytesPerSec":104.59025218435545,"startupDelaySec":-168.589,"processingDelaySec":72.101,"shutdownDelaySec":71.993,"timeDilation":10.180545778293096,"errors":null,"jobId":null}
2018-11-19T21:20:00.390Z RUNNING Query7
2018-11-19T21:20:01.740Z no activity
2018-11-19T21:20:16.838Z RUNNING Query7
2018-11-19T21:20:17.974Z no activity
2018-11-19T21:20:33.044Z RUNNING Query7
2018-11-19T21:20:34.434Z no activity
2018-11-19T21:20:49.522Z RUNNING Query7
2018-11-19T21:20:51.450Z no activity
2018-11-19T21:21:06.542Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 5,5,main]) completed. Took 5 mins 53.492 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 32s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/dtqwrnjnqkjwy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1084
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1084/display/redirect>
------------------------------------------
[...truncated 249.25 KB...]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-WrzthlNnJHVoj7ykTxMTaA.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-WrzthlNnJHVoj7ykTxMTaA.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 19, 2018 6:04:10 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-WrzthlNnJHVoj7ykTxMTaA.jar
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 19, 2018 6:04:13 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 19, 2018 6:04:14 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 19, 2018 6:04:14 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 19, 2018 6:04:14 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 19, 2018 6:04:14 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 19, 2018 6:04:14 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 6:04:14 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88185 bytes, hash cSFuJ1KF_a-sJn-dl71JYA> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-cSFuJ1KF_a-sJn-dl71JYA.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 19, 2018 6:04:15 PM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-19_10_04_14-17036458763679961965
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_10_04_14-17036458763679961965?project=apache-beam-testing
Nov 19, 2018 6:04:15 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_10_04_14-17036458763679961965
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T18:04:14.698Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:17.685Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:24.140Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:24.606Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:24.691Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:24.727Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:24.911Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:24.956Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#6044f02ac45a6f71
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.007Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.063Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.111Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.158Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.206Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.248Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.294Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.328Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.382Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.428Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.469Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.513Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.564Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.606Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.659Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.707Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.752Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.809Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.853Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.893Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.935Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:25.977Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:26.019Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:26.082Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:26.133Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:26.175Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:26.218Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:26.264Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:26.309Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:26.349Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:26.388Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:26.436Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:27.012Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:27.046Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:27.086Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:27.107Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 6:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:27.428Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 19, 2018 6:04:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:04:39.022Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 6:05:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:05:08.422Z: Workers have started successfully.
Nov 19, 2018 6:05:21 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:05:19.722Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 6:05:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:05:38.667Z: Workers have started successfully.
Nov 19, 2018 6:06:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:06:27.323Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 19, 2018 6:06:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:06:27.401Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 19, 2018 6:06:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:06:29.043Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 19, 2018 6:06:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:06:29.132Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 19, 2018 6:06:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:06:31.605Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 19, 2018 6:06:33 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:06:31.673Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 19, 2018 6:06:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:06:34.969Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 19, 2018 6:06:36 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:06:35.165Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 19, 2018 6:07:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:07:00.055Z: Cleaning up.
Nov 19, 2018 6:07:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T18:07:00.152Z: Stopping worker pool...
Nov 19, 2018 6:08:15 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-19T18:08:15.805Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-19T18:08:15.841Z RUNNING Query7
2018-11-19T18:08:17.050Z new perf {"snapshots":null,"runtimeSec":95.301,"numEvents":10000000,"eventsPerSec":104930.69327709048,"eventBytesPerSec":1.3468535450834723E7,"numResults":100,"resultsPerSec":1.0493069327709048,"resultBytesPerSec":106.01147941784451,"startupDelaySec":-171.583,"processingDelaySec":73.218,"shutdownDelaySec":76.282,"timeDilation":10.318884376869077,"errors":null,"jobId":null}
2018-11-19T18:08:32.117Z RUNNING Query7
2018-11-19T18:08:33.227Z no activity
2018-11-19T18:08:48.287Z RUNNING Query7
2018-11-19T18:08:49.516Z no activity
2018-11-19T18:09:04.580Z RUNNING Query7
2018-11-19T18:09:05.880Z no activity
2018-11-19T18:09:20.920Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 5 mins 29.315 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 14s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/6vbcrvzdrjwy2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1083
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1083/display/redirect?page=changes>
Changes:
[mxm] [BEAM-4681] Add support for portable timers in Flink batch mode
[mxm] [BEAM-4681] Use CoderUtils to decode key
[mxm] [BEAM-867] Support set/delete of timers by ID in FlinkRunner
[mxm] [BEAM-4681] Enable Java portable ValidatesRunner timer tests
[mxm] [BEAM-4681] Enable Python ValidatesRunner timer tests
[mxm] [BEAM-4681] Simplify by replacing the two timer keys with a single one
------------------------------------------
[...truncated 379.09 KB...]
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding NexmarkUtils.GenerateSideInputData/MapElements/Map as step s3
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Monitor as step s4
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Snoop as step s5
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) as step s6
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s7
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike) as step s8
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize as step s9
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize) as step s10
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys as step s11
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) as step s12
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections as step s13
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView as step s14
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) as step s15
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/AsBid as step s16
Nov 19, 2018 4:09:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles as step s17
Nov 19, 2018 4:09:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Debug as step s18
Nov 19, 2018 4:09:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Stamp as step s19
Nov 19, 2018 4:09:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.Format as step s20
Nov 19, 2018 4:09:07 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.DevNull as step s21
Nov 19, 2018 4:09:07 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 4:09:07 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <58933 bytes, hash 1XV7Xl1Ldd0lKt6cqVtoDg> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-1XV7Xl1Ldd0lKt6cqVtoDg.pb
Nov 19, 2018 4:09:07 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-185dmBDT2hA3f6JXCdfhuQ.jar
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 19, 2018 4:09:08 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-185dmBDT2hA3f6JXCdfhuQ.jar
Nov 19, 2018 4:09:08 PM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-19_08_09_07-10491805752574827508
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_08_09_07-10491805752574827508?project=apache-beam-testing
Nov 19, 2018 4:09:08 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_08_09_07-10491805752574827508
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 117 files cached, 28 files newly uploaded
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 4:09:11 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88203 bytes, hash XIPCnsKzhin97x-Qt5FIyA> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-XIPCnsKzhin97x-Qt5FIyA.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T16:09:07.674Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:10.746Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:15.768Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.374Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.533Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.563Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.650Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.686Z: Fusing consumer JoinToFiles/JoinToFiles.Monitor into JoinToFiles.ReadBounded
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.721Z: Fusing consumer JoinToFiles/JoinToFiles.Snoop into JoinToFiles/JoinToFiles.Monitor
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.770Z: Fusing consumer JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) into JoinToFiles/JoinToFiles.Snoop
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.811Z: Fusing consumer JoinToFiles/JoinToFiles/justBids/AsBid into JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous)
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.857Z: Fusing consumer JoinToFiles.Format into JoinToFiles/JoinToFiles.Stamp
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.892Z: Fusing consumer JoinToFiles.DevNull into JoinToFiles.Format
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.936Z: Fusing consumer JoinToFiles/JoinToFiles.Stamp into JoinToFiles/JoinToFiles.Debug
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:16.978Z: Fusing consumer JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles into JoinToFiles/JoinToFiles/justBids/AsBid
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.012Z: Fusing consumer JoinToFiles/JoinToFiles.Debug into JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.059Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Write into JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.091Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Read
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.134Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Read
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.214Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into NexmarkUtils.GenerateSideInputData/MapElements/Map
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.267Z: Fusing consumer NexmarkUtils.GenerateSideInputData/MapElements/Map into NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.313Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.354Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Write into JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.388Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.906Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Create
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:17.961Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:18.008Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Create
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:18.020Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 4:09:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:18.315Z: Executing operation NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)+NexmarkUtils.GenerateSideInputData/MapElements/Map+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 19, 2018 4:09:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:29.299Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 4:09:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:52.334Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 19, 2018 4:09:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:52.369Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 19, 2018 4:10:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:58.641Z: Workers have started successfully.
Nov 19, 2018 4:10:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:09:58.679Z: Workers have started successfully.
Nov 19, 2018 4:10:04 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:03.410Z: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s).
Nov 19, 2018 4:10:04 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:03.447Z: Resized worker pool to 2, though goal was 4. This could be a quota issue.
Nov 19, 2018 4:10:11 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:09.396Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 4:10:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:18.521Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 19, 2018 4:10:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:18.602Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Write+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Write
Nov 19, 2018 4:10:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:24.990Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Close
Nov 19, 2018 4:10:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:25.021Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Close
Nov 19, 2018 4:10:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:25.052Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Read+JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey)
Nov 19, 2018 4:10:25 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:25.095Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Read+JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize)
Nov 19, 2018 4:10:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:31.633Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections
Nov 19, 2018 4:10:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:31.886Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView
Nov 19, 2018 4:10:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:10:32.073Z: Executing operation JoinToFiles.ReadBounded+JoinToFiles/JoinToFiles.Monitor+JoinToFiles/JoinToFiles.Snoop+JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous)+JoinToFiles/JoinToFiles/justBids/AsBid+JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles+JoinToFiles/JoinToFiles.Debug+JoinToFiles/JoinToFiles.Stamp+JoinToFiles.Format+JoinToFiles.DevNull
Nov 19, 2018 4:11:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:11:28.316Z: Cleaning up.
Nov 19, 2018 4:11:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T16:11:28.436Z: Stopping worker pool...
Nov 19, 2018 4:13:08 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-19T16:13:08.390Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-19T16:13:08.429Z RUNNING JoinToFiles
2018-11-19T16:13:09.509Z new perf {"snapshots":null,"runtimeSec":53.893,"numEvents":10000000,"eventsPerSec":185552.85473067002,"eventBytesPerSec":2.3816913087042842E7,"numResults":9200000,"resultsPerSec":170708.62635221644,"resultBytesPerSec":6084868.071920287,"startupDelaySec":-154.18,"processingDelaySec":0.809,"shutdownDelaySec":100.287,"timeDilation":18.55435770879335,"errors":null,"jobId":null}
2018-11-19T16:13:24.545Z RUNNING JoinToFiles
2018-11-19T16:13:25.535Z no activity
2018-11-19T16:13:40.574Z RUNNING JoinToFiles
2018-11-19T16:13:41.597Z no activity
2018-11-19T16:13:56.636Z RUNNING JoinToFiles
2018-11-19T16:13:57.632Z no activity
2018-11-19T16:14:12.701Z DONE JoinToFiles
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 4,5,main]) completed. Took 5 mins 36.389 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 16s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/l4jmdtfgdeta4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1082
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1082/display/redirect?page=changes>
Changes:
[echauchot] [BEAM-5969] Add BOUNDED_SIDE_INPUT_JOIN new Query to Beam website
------------------------------------------
[...truncated 251.60 KB...]
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-TcaOL84XDQbz_qXK48pSNg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-TcaOL84XDQbz_qXK48pSNg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 19, 2018 3:25:57 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-TcaOL84XDQbz_qXK48pSNg.jar
Nov 19, 2018 3:26:00 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 3:26:01 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88176 bytes, hash IoDa3VhZjVHkM_WjV5P3dA> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-IoDa3VhZjVHkM_WjV5P3dA.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Submitted job: 2018-11-19_07_26_01-2613775342329271758
Nov 19, 2018 3:26:02 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_07_26_01-2613775342329271758?project=apache-beam-testing
Nov 19, 2018 3:26:02 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_07_26_01-2613775342329271758
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T15:26:01.817Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:04.421Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:10.132Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:10.615Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:10.696Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:10.750Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:10.945Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:10.984Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#6044f02ac45a6f71
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.034Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.068Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.112Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.152Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.194Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.242Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.290Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.333Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.366Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.410Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.449Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.497Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.532Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.564Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.604Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.640Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.664Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.707Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.755Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.792Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.830Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.868Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.916Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:11.961Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.010Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.058Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.105Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.143Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.177Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.217Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.254Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.290Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.715Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.754Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.786Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:12.812Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 3:26:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:13.044Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 19, 2018 3:26:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:23.714Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 3:26:49 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:49.478Z: Workers have started successfully.
Nov 19, 2018 3:26:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:52.427Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 19, 2018 3:26:52 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:52.463Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 19, 2018 3:26:57 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:56.432Z: Workers have started successfully.
Nov 19, 2018 3:27:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:58.259Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 19, 2018 3:27:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:26:58.307Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 19, 2018 3:27:12 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:27:09.491Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 3:28:11 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:28:09.841Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 19, 2018 3:28:11 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:28:09.924Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 19, 2018 3:28:11 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:28:11.476Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 19, 2018 3:28:11 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:28:11.553Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 19, 2018 3:28:14 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:28:12.906Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 19, 2018 3:28:14 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:28:12.976Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 19, 2018 3:28:17 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:28:16.639Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 19, 2018 3:28:17 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:28:16.811Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 19, 2018 3:28:44 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:28:42.705Z: Cleaning up.
Nov 19, 2018 3:28:44 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T15:28:42.803Z: Stopping worker pool...
Nov 19, 2018 3:30:02 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-19T15:30:02.676Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-19T15:30:02.720Z RUNNING Query7
2018-11-19T15:30:03.842Z new perf {"snapshots":null,"runtimeSec":98.609,"numEvents":10000000,"eventsPerSec":101410.62174852194,"eventBytesPerSec":1.3016711425934752E7,"numResults":100,"resultsPerSec":1.0141062174852196,"resultBytesPerSec":102.45515115253173,"startupDelaySec":-179.357,"processingDelaySec":75.364,"shutdownDelaySec":80.748,"timeDilation":9.972720542749649,"errors":null,"jobId":null}
2018-11-19T15:30:18.881Z RUNNING Query7
2018-11-19T15:30:19.879Z no activity
2018-11-19T15:30:34.941Z RUNNING Query7
2018-11-19T15:30:36.015Z no activity
2018-11-19T15:30:51.080Z RUNNING Query7
2018-11-19T15:30:52.179Z no activity
2018-11-19T15:31:07.238Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':',5,main]) completed. Took 5 mins 36.231 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 23s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/oecadoswdcwik
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1081
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1081/display/redirect?page=changes>
Changes:
[robbe.sneyders] Fix unorderable types in Python 3
[robbe.sneyders] Check equality of collections by sorting first, fall back to slower
------------------------------------------
[...truncated 270.81 KB...]
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-LP8Z5DZBYSls1Ii6SrBICQ.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-LP8Z5DZBYSls1Ii6SrBICQ.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 19, 2018 1:03:01 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-LP8Z5DZBYSls1Ii6SrBICQ.jar
Nov 19, 2018 1:03:05 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-LP8Z5DZBYSls1Ii6SrBICQ.jar
Nov 19, 2018 1:03:08 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 1:03:09 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88175 bytes, hash cjr_xFNiODh5fwiFrxbdfQ> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-cjr_xFNiODh5fwiFrxbdfQ.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 19, 2018 1:03:10 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_05_03_09-12742929081357882146?project=apache-beam-testing
Submitted job: 2018-11-19_05_03_09-12742929081357882146
Nov 19, 2018 1:03:10 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_05_03_09-12742929081357882146
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T13:03:09.798Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:12.721Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:17.196Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:17.689Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:17.774Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:17.860Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.056Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.102Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.151Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.197Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.243Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.291Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.339Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.388Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.432Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.474Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.513Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.557Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.594Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.642Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.689Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.738Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.785Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.831Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.879Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.923Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:18.968Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.027Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.066Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.103Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.146Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.196Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.230Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.260Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.303Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.369Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.481Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.546Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.637Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:19.699Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:20.203Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:20.251Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:20.297Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:20.309Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 1:03:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:20.599Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 19, 2018 1:03:31 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:29.508Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 1:03:56 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:56.063Z: Workers have started successfully.
Nov 19, 2018 1:03:58 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:57.429Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 19, 2018 1:03:58 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:03:57.471Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 19, 2018 1:04:12 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:04:09.661Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 1:04:28 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:04:26.258Z: Workers have started successfully.
Nov 19, 2018 1:05:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:05:12.306Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 19, 2018 1:05:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:05:12.422Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 19, 2018 1:05:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:05:14.145Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 19, 2018 1:05:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:05:14.251Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 19, 2018 1:05:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:05:14.684Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 19, 2018 1:05:16 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:05:14.769Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 19, 2018 1:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:05:17.934Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 19, 2018 1:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:05:18.131Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 19, 2018 1:05:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:05:40.665Z: Cleaning up.
Nov 19, 2018 1:05:42 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T13:05:40.806Z: Stopping worker pool...
Nov 19, 2018 1:07:10 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-19T13:07:10.471Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-19T13:07:10.515Z RUNNING Query7
2018-11-19T13:07:11.622Z new perf {"snapshots":null,"runtimeSec":90.665,"numEvents":10000000,"eventsPerSec":110296.145149727,"eventBytesPerSec":1.415722601886064E7,"numResults":100,"resultsPerSec":1.10296145149727,"resultBytesPerSec":111.43219544476919,"startupDelaySec":-180.886,"processingDelaySec":70.353,"shutdownDelaySec":90.221,"timeDilation":10.846522914024154,"errors":null,"jobId":null}
2018-11-19T13:07:26.667Z RUNNING Query7
2018-11-19T13:07:27.665Z no activity
2018-11-19T13:07:42.700Z RUNNING Query7
2018-11-19T13:07:43.678Z no activity
2018-11-19T13:07:58.720Z RUNNING Query7
2018-11-19T13:07:59.791Z no activity
2018-11-19T13:08:14.827Z RUNNING Query7
2018-11-19T13:08:15.882Z no activity
2018-11-19T13:08:30.922Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':',5,main]) completed. Took 5 mins 49.95 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 23s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/7f63pogk7ireq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1080
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1080/display/redirect>
------------------------------------------
[...truncated 260.00 KB...]
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-AMcqIIX7HBb2ui2k_IuGwA.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-AMcqIIX7HBb2ui2k_IuGwA.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Nov 19, 2018 12:04:55 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackageWithRetry
WARNING: Upload attempt failed, sleeping before retrying staging of package: <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar>
java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-AMcqIIX7HBb2ui2k_IuGwA.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-AMcqIIX7HBb2ui2k_IuGwA.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:432)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:287)
at org.apache.beam.runners.dataflow.util.PackageUtil.$closeResource(PackageUtil.java:260)
at org.apache.beam.runners.dataflow.util.PackageUtil.tryStagePackage(PackageUtil.java:260)
at org.apache.beam.runners.dataflow.util.PackageUtil.tryStagePackageWithRetry(PackageUtil.java:203)
at org.apache.beam.runners.dataflow.util.PackageUtil.stagePackageSynchronously(PackageUtil.java:187)
at org.apache.beam.runners.dataflow.util.PackageUtil.lambda$stagePackage$1(PackageUtil.java:171)
at org.apache.beam.sdk.util.MoreFutures.lambda$supplyAsync$0(MoreFutures.java:104)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-AMcqIIX7HBb2ui2k_IuGwA.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-AMcqIIX7HBb2ui2k_IuGwA.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 19, 2018 12:05:02 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-AMcqIIX7HBb2ui2k_IuGwA.jar
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.ReadBounded as step s1
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource) as step s2
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding NexmarkUtils.GenerateSideInputData/MapElements/Map as step s3
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Monitor as step s4
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Snoop as step s5
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) as step s6
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s7
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike) as step s8
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize as step s9
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize) as step s10
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys as step s11
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) as step s12
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections as step s13
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView as step s14
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) as step s15
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/AsBid as step s16
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles as step s17
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Debug as step s18
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Stamp as step s19
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.Format as step s20
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.DevNull as step s21
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 12:05:06 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <58918 bytes, hash XCOGjbMs4ILUNzTZP8b-AQ> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-XCOGjbMs4ILUNzTZP8b-AQ.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 19, 2018 12:05:08 PM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-19_04_05_07-4988522667580810347
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_04_05_07-4988522667580810347?project=apache-beam-testing
Nov 19, 2018 12:05:08 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_04_05_07-4988522667580810347
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T12:05:07.348Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:10.099Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:15.816Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.299Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.494Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.526Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.617Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.656Z: Fusing consumer JoinToFiles/JoinToFiles.Monitor into JoinToFiles.ReadBounded
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.704Z: Fusing consumer JoinToFiles/JoinToFiles.Snoop into JoinToFiles/JoinToFiles.Monitor
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.753Z: Fusing consumer JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) into JoinToFiles/JoinToFiles.Snoop
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.802Z: Fusing consumer JoinToFiles/JoinToFiles/justBids/AsBid into JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous)
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.853Z: Fusing consumer JoinToFiles.Format into JoinToFiles/JoinToFiles.Stamp
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.895Z: Fusing consumer JoinToFiles.DevNull into JoinToFiles.Format
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.943Z: Fusing consumer JoinToFiles/JoinToFiles.Stamp into JoinToFiles/JoinToFiles.Debug
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:16.984Z: Fusing consumer JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles into JoinToFiles/JoinToFiles/justBids/AsBid
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.023Z: Fusing consumer JoinToFiles/JoinToFiles.Debug into JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.069Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Write into JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.118Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Read
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.162Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Read
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.209Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into NexmarkUtils.GenerateSideInputData/MapElements/Map
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.254Z: Fusing consumer NexmarkUtils.GenerateSideInputData/MapElements/Map into NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.284Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.329Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Write into JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.372Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.866Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Create
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.907Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.946Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Create
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:17.981Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 12:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:18.229Z: Executing operation NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)+NexmarkUtils.GenerateSideInputData/MapElements/Map+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 19, 2018 12:05:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:05:39.139Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 12:06:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:08.405Z: Workers have started successfully.
Nov 19, 2018 12:06:14 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:12.705Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 19, 2018 12:06:14 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:12.738Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 19, 2018 12:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:18.212Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 12:06:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:27.341Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 19, 2018 12:06:27 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:27.479Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Write+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Write
Nov 19, 2018 12:06:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:33.710Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Close
Nov 19, 2018 12:06:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:33.761Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Close
Nov 19, 2018 12:06:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:33.811Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Read+JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey)
Nov 19, 2018 12:06:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:33.861Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Read+JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize)
Nov 19, 2018 12:06:40 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:39.566Z: Workers have started successfully.
Nov 19, 2018 12:06:44 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:42.807Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections
Nov 19, 2018 12:06:44 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:43.124Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView
Nov 19, 2018 12:06:44 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:06:43.384Z: Executing operation JoinToFiles.ReadBounded+JoinToFiles/JoinToFiles.Monitor+JoinToFiles/JoinToFiles.Snoop+JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous)+JoinToFiles/JoinToFiles/justBids/AsBid+JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles+JoinToFiles/JoinToFiles.Debug+JoinToFiles/JoinToFiles.Stamp+JoinToFiles.Format+JoinToFiles.DevNull
Nov 19, 2018 12:07:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:07:42.395Z: Cleaning up.
Nov 19, 2018 12:07:43 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T12:07:42.528Z: Stopping worker pool...
Nov 19, 2018 12:09:08 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-19T12:09:08.249Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-19T12:09:08.280Z RUNNING JoinToFiles
2018-11-19T12:09:09.368Z new perf {"snapshots":null,"runtimeSec":56.567,"numEvents":10000000,"eventsPerSec":176781.51572471583,"eventBytesPerSec":2.2691054802269876E7,"numResults":9200000,"resultsPerSec":162638.99446673857,"resultBytesPerSec":5797227.977442679,"startupDelaySec":-143.625,"processingDelaySec":0.86,"shutdownDelaySec":87.058,"timeDilation":17.67726766489296,"errors":null,"jobId":null}
2018-11-19T12:09:24.418Z RUNNING JoinToFiles
2018-11-19T12:09:25.580Z no activity
2018-11-19T12:09:40.623Z RUNNING JoinToFiles
2018-11-19T12:09:41.684Z no activity
2018-11-19T12:09:56.726Z RUNNING JoinToFiles
2018-11-19T12:09:57.854Z no activity
2018-11-19T12:10:12.898Z DONE JoinToFiles
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Daemon worker,5,main]) completed. Took 5 mins 34.205 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 57s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/jbl77xqfxg2w6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1079
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1079/display/redirect?page=changes>
Changes:
[robertwb] Implement profiling for Python DirectRunner and SDK Harness.
------------------------------------------
[...truncated 275.39 KB...]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Nov 19, 2018 10:01:49 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackageWithRetry
WARNING: Upload attempt failed, sleeping before retrying staging of package: <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar>
java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-TayvjSPHqqmk9WZIHinJhg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-TayvjSPHqqmk9WZIHinJhg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:432)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:287)
at org.apache.beam.runners.dataflow.util.PackageUtil.$closeResource(PackageUtil.java:260)
at org.apache.beam.runners.dataflow.util.PackageUtil.tryStagePackage(PackageUtil.java:260)
at org.apache.beam.runners.dataflow.util.PackageUtil.tryStagePackageWithRetry(PackageUtil.java:203)
at org.apache.beam.runners.dataflow.util.PackageUtil.stagePackageSynchronously(PackageUtil.java:187)
at org.apache.beam.runners.dataflow.util.PackageUtil.lambda$stagePackage$1(PackageUtil.java:171)
at org.apache.beam.sdk.util.MoreFutures.lambda$supplyAsync$0(MoreFutures.java:104)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-TayvjSPHqqmk9WZIHinJhg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-TayvjSPHqqmk9WZIHinJhg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 19, 2018 10:01:55 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-TayvjSPHqqmk9WZIHinJhg.jar
Nov 19, 2018 10:01:57 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-TayvjSPHqqmk9WZIHinJhg.jar
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3.ReadBounded as step s1
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3.Monitor as step s2
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3.Snoop as step s3
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Window.Into()/Window.Assign as step s4
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/justNewAuctions/IsNewAuction/ParDo(Anonymous) as step s5
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/justNewAuctions/AsAuction as step s6
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.InCategory/ParDo(Anonymous) as step s7
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/AuctionBySeller as step s8
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/justNewPersons/IsNewPerson/ParDo(Anonymous) as step s9
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/justNewPersons/AsPerson as step s10
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.InState/ParDo(Anonymous) as step s11
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/PersonById as step s12
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/CoGroupByKey/MakeUnionTable0 as step s13
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/CoGroupByKey/MakeUnionTable1 as step s14
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/CoGroupByKey/Flatten as step s15
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/CoGroupByKey/GBK as step s16
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/CoGroupByKey/ConstructCoGbkResultFn as step s17
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/ReifyWindows as step s18
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys as step s19
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.Join/ParDo(BatchStateful) as step s20
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3/Query3.Project as step s21
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3.Debug as step s22
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3/Query3.Stamp as step s23
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3.Format as step s24
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query3.DevNull as step s25
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 10:02:01 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <65272 bytes, hash grKeLsdL3_hUklqH3hU2lQ> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-grKeLsdL3_hUklqH3hU2lQ.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 19, 2018 10:02:02 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-19_02_02_02-16030366142429415685
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_02_02_02-16030366142429415685?project=apache-beam-testing
Nov 19, 2018 10:02:02 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_02_02_02-16030366142429415685
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T10:02:02.237Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:04.883Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:08.651Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.146Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.318Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.363Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.520Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.572Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#bed3177b52ae68e
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.612Z: Fusing unzipped copy of Query3/Query3/CoGroupByKey/GBK/Reify, through flatten Query3/Query3/CoGroupByKey/Flatten, into producer Query3/Query3/CoGroupByKey/MakeUnionTable0
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.665Z: Fusing consumer Query3/Query3.Stamp into Query3/Query3.Debug
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.713Z: Fusing consumer Query3.Format into Query3/Query3.Stamp
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.774Z: Unzipping flatten s15-u18 for input s16-reify-value0-c16
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.822Z: Fusing unzipped copy of Query3/Query3/CoGroupByKey/GBK/Write, through flatten s15-u18, into producer Query3/Query3/CoGroupByKey/GBK/Reify
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.874Z: Fusing consumer Query3.DevNull into Query3.Format
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.908Z: Fusing consumer Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Write into Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/ReifyWindows
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.960Z: Fusing consumer Query3/Query3/Query3.Project into Query3/Query3/Query3.Join/ParDo(BatchStateful)
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:09.998Z: Fusing consumer Query3/Query3.Debug into Query3/Query3/Query3.Project
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.061Z: Fusing consumer Query3/Query3/Query3.Join/ParDo(BatchStateful) into Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Read
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.120Z: Fusing consumer Query3/Query3/CoGroupByKey/GBK/GroupByWindow into Query3/Query3/CoGroupByKey/GBK/Read
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.161Z: Fusing consumer Query3/Query3/CoGroupByKey/ConstructCoGbkResultFn into Query3/Query3/CoGroupByKey/GBK/GroupByWindow
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.214Z: Fusing consumer Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/ReifyWindows into Query3/Query3/CoGroupByKey/ConstructCoGbkResultFn
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.261Z: Fusing consumer Query3/Query3/CoGroupByKey/GBK/Reify into Query3/Query3/CoGroupByKey/MakeUnionTable1
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.298Z: Fusing consumer Query3/Query3/CoGroupByKey/GBK/Write into Query3/Query3/CoGroupByKey/GBK/Reify
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.346Z: Fusing consumer Query3/Query3/justNewAuctions/IsNewAuction/ParDo(Anonymous) into Query3/Query3/Window.Into()/Window.Assign
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.392Z: Fusing consumer Query3/Query3/justNewPersons/IsNewPerson/ParDo(Anonymous) into Query3/Query3/Window.Into()/Window.Assign
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.439Z: Fusing consumer Query3/Query3/Query3.InCategory/ParDo(Anonymous) into Query3/Query3/justNewAuctions/AsAuction
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.486Z: Fusing consumer Query3/Query3.Snoop into Query3/Query3.Monitor
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.526Z: Fusing consumer Query3/Query3/PersonById into Query3/Query3/Query3.InState/ParDo(Anonymous)
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.572Z: Fusing consumer Query3/Query3/CoGroupByKey/MakeUnionTable1 into Query3/Query3/PersonById
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.621Z: Fusing consumer Query3/Query3/justNewAuctions/AsAuction into Query3/Query3/justNewAuctions/IsNewAuction/ParDo(Anonymous)
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.668Z: Fusing consumer Query3/Query3/Query3.InState/ParDo(Anonymous) into Query3/Query3/justNewPersons/AsPerson
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.708Z: Fusing consumer Query3/Query3/justNewPersons/AsPerson into Query3/Query3/justNewPersons/IsNewPerson/ParDo(Anonymous)
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.755Z: Fusing consumer Query3/Query3/AuctionBySeller into Query3/Query3/Query3.InCategory/ParDo(Anonymous)
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.803Z: Fusing consumer Query3/Query3.Monitor into Query3.ReadBounded
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.851Z: Fusing consumer Query3/Query3/Window.Into()/Window.Assign into Query3/Query3.Snoop
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:10.900Z: Fusing consumer Query3/Query3/CoGroupByKey/MakeUnionTable0 into Query3/Query3/AuctionBySeller
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:11.416Z: Executing operation Query3/Query3/CoGroupByKey/GBK/Create
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:11.464Z: Executing operation Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Create
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:11.512Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 10:02:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:11.741Z: Executing operation Query3.ReadBounded+Query3/Query3.Monitor+Query3/Query3.Snoop+Query3/Query3/Window.Into()/Window.Assign+Query3/Query3/justNewAuctions/IsNewAuction/ParDo(Anonymous)+Query3/Query3/justNewPersons/IsNewPerson/ParDo(Anonymous)+Query3/Query3/justNewAuctions/AsAuction+Query3/Query3/Query3.InCategory/ParDo(Anonymous)+Query3/Query3/justNewPersons/AsPerson+Query3/Query3/Query3.InState/ParDo(Anonymous)+Query3/Query3/PersonById+Query3/Query3/CoGroupByKey/MakeUnionTable1+Query3/Query3/CoGroupByKey/GBK/Reify+Query3/Query3/AuctionBySeller+Query3/Query3/CoGroupByKey/MakeUnionTable0+Query3/Query3/CoGroupByKey/GBK/Reify+Query3/Query3/CoGroupByKey/GBK/Session/Flatten+Query3/Query3/CoGroupByKey/GBK/Write
Nov 19, 2018 10:02:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:23.587Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 10:02:50 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:49.455Z: Workers have started successfully.
Nov 19, 2018 10:02:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:02:57.862Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 10:03:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:03:19.882Z: Workers have started successfully.
Nov 19, 2018 10:03:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:03:43.983Z: Executing operation Query3/Query3/CoGroupByKey/GBK/Close
Nov 19, 2018 10:03:46 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:03:44.061Z: Executing operation Query3/Query3/CoGroupByKey/GBK/Read+Query3/Query3/CoGroupByKey/GBK/GroupByWindow+Query3/Query3/CoGroupByKey/ConstructCoGbkResultFn+Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/ReifyWindows+Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Write
Nov 19, 2018 10:03:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:03:48.586Z: Executing operation Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Close
Nov 19, 2018 10:03:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:03:48.668Z: Executing operation Query3/Query3/Query3.Join/BatchStatefulParDoOverrides.GbkBeforeStatefulParDo/PartitionKeys/Read+Query3/Query3/Query3.Join/ParDo(BatchStateful)+Query3/Query3/Query3.Project+Query3/Query3.Debug+Query3/Query3.Stamp+Query3.Format+Query3.DevNull
Nov 19, 2018 10:03:53 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:03:53.058Z: Cleaning up.
Nov 19, 2018 10:03:53 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:03:53.138Z: Stopping worker pool...
Nov 19, 2018 10:05:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:05:46.519Z: Autoscaling: Resized worker pool from 4 to 0.
Nov 19, 2018 10:05:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T10:05:46.573Z: Worker pool stopped.
Nov 19, 2018 10:05:53 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
INFO: Job 2018-11-19_02_02_02-16030366142429415685 finished with status DONE.
2018-11-19T10:05:53.716Z Waiting for main pipeline to 'finish'
2018-11-19T10:05:53.716Z DONE Query3
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 6,5,main]) completed. Took 4 mins 20.904 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 7m 46s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/dwia4atd3lblo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1078
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1078/display/redirect?page=changes>
Changes:
[lukasz.gajowy] [BEAM-6076] Fetch only required fields for BigQuery Table
------------------------------------------
[...truncated 276.82 KB...]
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-OBgJSP2NNoyihGY9r3ESWg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 19, 2018 9:52:25 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-OBgJSP2NNoyihGY9r3ESWg.jar
Nov 19, 2018 9:52:28 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-OBgJSP2NNoyihGY9r3ESWg.jar
Nov 19, 2018 9:52:31 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-OBgJSP2NNoyihGY9r3ESWg.jar
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 19, 2018 9:52:33 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 9:52:34 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88177 bytes, hash oMgfbu2Q-xjgi7TAYwgSSg> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-oMgfbu2Q-xjgi7TAYwgSSg.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 19, 2018 9:52:35 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-19_01_52_34-7362580033603177336
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-19_01_52_34-7362580033603177336?project=apache-beam-testing
Nov 19, 2018 9:52:35 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-19_01_52_34-7362580033603177336
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T09:52:34.588Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:37.306Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:44.093Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:44.626Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:44.703Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:44.739Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:44.898Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:44.937Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:44.976Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:44.999Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.031Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.069Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.111Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.148Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.184Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.224Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.258Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.298Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.336Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.373Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.413Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.448Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.500Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.536Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.571Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.603Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.641Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.678Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.716Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.756Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.783Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.812Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.850Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.886Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.925Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.962Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:45.999Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:46.037Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:46.065Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:46.101Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:46.513Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:46.541Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:46.579Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:46.589Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 9:52:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:46.814Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 19, 2018 9:52:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:52:58.574Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 9:53:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:53:24.552Z: Workers have started successfully.
Nov 19, 2018 9:53:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:53:27.538Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 19, 2018 9:53:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:53:27.571Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 19, 2018 9:53:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:53:32.139Z: Workers have started successfully.
Nov 19, 2018 9:53:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:53:33.457Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 9:54:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:54:46.404Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 19, 2018 9:54:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:54:46.485Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 19, 2018 9:54:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:54:48.086Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 19, 2018 9:54:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:54:48.156Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 19, 2018 9:54:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:54:49.599Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 19, 2018 9:54:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:54:49.670Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 19, 2018 9:54:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:54:52.718Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 19, 2018 9:54:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:54:52.891Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 19, 2018 9:55:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:55:16.712Z: Cleaning up.
Nov 19, 2018 9:55:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T09:55:16.814Z: Stopping worker pool...
Nov 19, 2018 9:56:35 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
WARNING: No terminal state was returned. State value RUNNING
2018-11-19T09:56:35.402Z Waiting for main pipeline to 'finish'
2018-11-19T09:56:35.440Z RUNNING Query7
2018-11-19T09:56:36.469Z new perf {"snapshots":null,"runtimeSec":98.223,"numEvents":10000000,"eventsPerSec":101809.1485700905,"eventBytesPerSec":1.3067864929802593E7,"numResults":100,"resultsPerSec":1.0180914857009051,"resultBytesPerSec":102.85778280036244,"startupDelaySec":-177.32,"processingDelaySec":76.336,"shutdownDelaySec":79.097,"timeDilation":10.0119116703827,"errors":null,"jobId":null}
2018-11-19T09:56:51.510Z RUNNING Query7
2018-11-19T09:56:52.556Z no activity
2018-11-19T09:57:07.597Z RUNNING Query7
2018-11-19T09:57:08.549Z no activity
2018-11-19T09:57:23.585Z RUNNING Query7
2018-11-19T09:57:24.481Z no activity
2018-11-19T09:57:39.510Z RUNNING Query7
2018-11-19T09:57:40.482Z no activity
2018-11-19T09:57:55.536Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 7,5,main]) completed. Took 5 mins 48.081 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 10s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/wcay2shgzh2ce
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1077
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1077/display/redirect>
------------------------------------------
[...truncated 249.53 KB...]
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-v08U42TZqPfPk5N3z3LPTg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-v08U42TZqPfPk5N3z3LPTg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 19, 2018 6:04:11 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-v08U42TZqPfPk5N3z3LPTg.jar
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 28 files newly uploaded
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 19, 2018 6:04:14 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 6:04:15 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88177 bytes, hash nOlxKKYjQgb8RVUk8OFl2A> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-nOlxKKYjQgb8RVUk8OFl2A.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Submitted job: 2018-11-18_22_04_15-1171071426444180883
Nov 19, 2018 6:04:16 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-18_22_04_15-1171071426444180883?project=apache-beam-testing
Nov 19, 2018 6:04:16 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-18_22_04_15-1171071426444180883
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T06:04:15.553Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:18.660Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:25.808Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.420Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.496Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.532Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.666Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.701Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.737Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.777Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.810Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.846Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.885Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.923Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.963Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:26.998Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.040Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.069Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.104Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.143Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.181Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.223Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.260Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.295Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.340Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.368Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.404Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.442Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.478Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.515Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.559Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.591Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.621Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.658Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.698Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.734Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.773Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.808Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.846Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:27.886Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:28.341Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:28.375Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:28.410Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:28.425Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 6:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:28.708Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 19, 2018 6:04:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:04:47.661Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 6:05:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:05:16.569Z: Workers have started successfully.
Nov 19, 2018 6:05:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:05:23.284Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 6:05:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:05:48.186Z: Workers have started successfully.
Nov 19, 2018 6:06:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:06:34.002Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 19, 2018 6:06:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:06:34.102Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 19, 2018 6:06:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:06:35.856Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 19, 2018 6:06:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:06:35.965Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 19, 2018 6:06:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:06:37.650Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 19, 2018 6:06:37 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:06:37.765Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 19, 2018 6:06:42 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:06:41.264Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 19, 2018 6:06:42 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:06:41.461Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 19, 2018 6:07:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:07:07.281Z: Cleaning up.
Nov 19, 2018 6:07:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T06:07:07.446Z: Stopping worker pool...
Nov 19, 2018 6:08:16 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-19T06:08:16.403Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-19T06:08:16.444Z RUNNING Query7
2018-11-19T06:08:17.444Z new perf {"snapshots":null,"runtimeSec":95.744,"numEvents":10000000,"eventsPerSec":104445.1871657754,"eventBytesPerSec":1.3406217590658423E7,"numResults":100,"resultsPerSec":1.044451871657754,"resultBytesPerSec":105.52097259358288,"startupDelaySec":-165.31,"processingDelaySec":72.22,"shutdownDelaySec":69.566,"timeDilation":10.271139705882353,"errors":null,"jobId":null}
2018-11-19T06:08:32.534Z RUNNING Query7
2018-11-19T06:08:33.551Z no activity
2018-11-19T06:08:48.590Z RUNNING Query7
2018-11-19T06:08:49.579Z no activity
2018-11-19T06:09:04.625Z RUNNING Query7
2018-11-19T06:09:05.717Z no activity
2018-11-19T06:09:20.772Z RUNNING Query7
2018-11-19T06:09:21.797Z no activity
2018-11-19T06:09:36.847Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 4,5,main]) completed. Took 5 mins 47.517 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 25s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/5265vrvxkakuy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1076
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1076/display/redirect>
------------------------------------------
[...truncated 404.47 KB...]
Nov 19, 2018 12:04:10 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-rHivmErBfEwxK0X3kpNi0A.jar
Nov 19, 2018 12:04:10 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 28 files newly uploaded
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.ReadBounded as step s1
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource) as step s2
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding NexmarkUtils.GenerateSideInputData/MapElements/Map as step s3
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Monitor as step s4
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Snoop as step s5
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) as step s6
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s7
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike) as step s8
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize as step s9
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize) as step s10
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys as step s11
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) as step s12
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections as step s13
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView as step s14
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) as step s15
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/AsBid as step s16
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles as step s17
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Debug as step s18
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Stamp as step s19
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.Format as step s20
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.DevNull as step s21
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 12:04:11 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <58907 bytes, hash STrIFbq-2Ih5Xpgr_ndJwA> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-STrIFbq-2Ih5Xpgr_ndJwA.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-18_16_04_12-3040468906700270689
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-18_16_04_12-3040468906700270689?project=apache-beam-testing
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-18_16_04_12-3040468906700270689
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 28 files newly uploaded
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 19, 2018 12:04:13 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 19, 2018 12:04:14 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88177 bytes, hash C0YJRX-W2tKpcN6jGrRkKw> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-C0YJRX-W2tKpcN6jGrRkKw.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-19T00:04:12.082Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:14.979Z: Checking permissions granted to controller Service Account.
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:21.889Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:22.407Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:22.603Z: Expanding GroupByKey operations into optimizable parts.
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:22.651Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:22.748Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:22.788Z: Fusing consumer JoinToFiles/JoinToFiles.Monitor into JoinToFiles.ReadBounded
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:22.829Z: Fusing consumer JoinToFiles/JoinToFiles.Snoop into JoinToFiles/JoinToFiles.Monitor
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:22.877Z: Fusing consumer JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) into JoinToFiles/JoinToFiles.Snoop
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:22.925Z: Fusing consumer JoinToFiles/JoinToFiles/justBids/AsBid into JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous)
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:22.974Z: Fusing consumer JoinToFiles.Format into JoinToFiles/JoinToFiles.Stamp
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.022Z: Fusing consumer JoinToFiles.DevNull into JoinToFiles.Format
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.074Z: Fusing consumer JoinToFiles/JoinToFiles.Stamp into JoinToFiles/JoinToFiles.Debug
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.118Z: Fusing consumer JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles into JoinToFiles/JoinToFiles/justBids/AsBid
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.161Z: Fusing consumer JoinToFiles/JoinToFiles.Debug into JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.206Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Write into JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.251Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Read
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.295Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Read
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.343Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into NexmarkUtils.GenerateSideInputData/MapElements/Map
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.379Z: Fusing consumer NexmarkUtils.GenerateSideInputData/MapElements/Map into NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.425Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.472Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Write into JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:23.508Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:24.018Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Create
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:24.065Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:24.124Z: Starting 4 workers in us-central1-b...
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:24.124Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Create
Nov 19, 2018 12:04:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:24.393Z: Executing operation NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)+NexmarkUtils.GenerateSideInputData/MapElements/Map+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 19, 2018 12:04:35 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:04:34.165Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 19, 2018 12:05:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:02.295Z: Workers have started successfully.
Nov 19, 2018 12:05:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:02.800Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 19, 2018 12:05:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:02.845Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 19, 2018 12:05:10 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:08.319Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 19, 2018 12:05:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:20.060Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 19, 2018 12:05:21 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:20.152Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Write+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Write
Nov 19, 2018 12:05:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:27.419Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Close
Nov 19, 2018 12:05:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:27.469Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Close
Nov 19, 2018 12:05:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:27.517Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Read+JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey)
Nov 19, 2018 12:05:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:27.567Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Read+JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize)
Nov 19, 2018 12:05:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:33.462Z: Workers have started successfully.
Nov 19, 2018 12:05:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:34.462Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections
Nov 19, 2018 12:05:35 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:34.695Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView
Nov 19, 2018 12:05:35 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:05:34.923Z: Executing operation JoinToFiles.ReadBounded+JoinToFiles/JoinToFiles.Monitor+JoinToFiles/JoinToFiles.Snoop+JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous)+JoinToFiles/JoinToFiles/justBids/AsBid+JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles+JoinToFiles/JoinToFiles.Debug+JoinToFiles/JoinToFiles.Stamp+JoinToFiles.Format+JoinToFiles.DevNull
Nov 19, 2018 12:06:35 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:06:35.112Z: Cleaning up.
Nov 19, 2018 12:06:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-19T00:06:35.232Z: Stopping worker pool...
Nov 19, 2018 12:08:13 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-19T00:08:13.198Z Waiting for main pipeline to 'finish'WARNING: No terminal state was returned. State value RUNNING
2018-11-19T00:08:13.237Z RUNNING JoinToFiles
2018-11-19T00:08:14.247Z new perf {"snapshots":null,"runtimeSec":58.648,"numEvents":10000000,"eventsPerSec":170508.7982539899,"eventBytesPerSec":2.1885910806847632E7,"numResults":9200000,"resultsPerSec":156868.0943936707,"resultBytesPerSec":5591525.627472377,"startupDelaySec":-157.042,"processingDelaySec":0.717,"shutdownDelaySec":98.394,"timeDilation":17.050027281407722,"errors":null,"jobId":null}
2018-11-19T00:08:29.310Z RUNNING JoinToFiles
2018-11-19T00:08:30.535Z no activity
2018-11-19T00:08:45.575Z RUNNING JoinToFiles
2018-11-19T00:08:46.508Z no activity
2018-11-19T00:09:01.541Z RUNNING JoinToFiles
2018-11-19T00:09:02.674Z no activity
2018-11-19T00:09:17.722Z DONE JoinToFiles
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 8,5,main]) completed. Took 5 mins 33.989 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 11s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/3ma6ubw6m4pwg
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1075
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1075/display/redirect>
------------------------------------------
[...truncated 249.11 KB...]
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 18, 2018 6:04:03 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-hOLEWj9_NGsdxCeuIKY59g.jar
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 18, 2018 6:04:06 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88168 bytes, hash o2eBQXctceT-SL9Q8UtbGw> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-o2eBQXctceT-SL9Q8UtbGw.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Submitted job: 2018-11-18_10_04_07-61524305918729972
Nov 18, 2018 6:04:08 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-18_10_04_07-61524305918729972?project=apache-beam-testing
Nov 18, 2018 6:04:08 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-18_10_04_07-61524305918729972
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-18T18:04:07.426Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:10.754Z: Checking permissions granted to controller Service Account.
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:17.161Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:17.896Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:17.988Z: Expanding GroupByKey operations into optimizable parts.
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.034Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.229Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.264Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#5e4921c369c80b03
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.308Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.342Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.381Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.412Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.452Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.492Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.519Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.567Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.612Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.663Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.700Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.743Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.792Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.838Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.888Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.926Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:18.965Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.007Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.057Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.094Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.130Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.176Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.222Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.271Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.314Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.348Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.394Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.443Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.480Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.531Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.576Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:19.625Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:20.116Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:20.156Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:20.203Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:20.216Z: Starting 4 workers in us-central1-b...
Nov 18, 2018 6:04:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:20.533Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 18, 2018 6:04:39 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:04:37.542Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 18, 2018 6:05:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:05:05.973Z: Workers have started successfully.
Nov 18, 2018 6:05:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:05:12.312Z: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s).
Nov 18, 2018 6:05:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:05:12.363Z: Resized worker pool to 2, though goal was 4. This could be a quota issue.
Nov 18, 2018 6:05:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:05:17.870Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 18, 2018 6:05:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:05:17.911Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 18, 2018 6:05:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:05:23.486Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 18, 2018 6:05:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:05:36.182Z: Workers have started successfully.
Nov 18, 2018 6:06:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:06:27.228Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 18, 2018 6:06:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:06:27.320Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 18, 2018 6:06:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:06:29.032Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 18, 2018 6:06:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:06:29.124Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 18, 2018 6:06:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:06:30.602Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 18, 2018 6:06:30 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:06:30.698Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 18, 2018 6:06:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:06:34.550Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 18, 2018 6:06:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:06:34.802Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 18, 2018 6:07:04 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:07:03.036Z: Cleaning up.
Nov 18, 2018 6:07:04 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T18:07:03.223Z: Stopping worker pool...
Nov 18, 2018 6:08:08 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-18T18:08:08.734Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-18T18:08:08.774Z RUNNING Query7
2018-11-18T18:08:10.338Z new perf {"snapshots":null,"runtimeSec":103.697,"numEvents":10000000,"eventsPerSec":96434.8052499108,"eventBytesPerSec":1.237803308678168E7,"numResults":100,"resultsPerSec":0.9643480524991079,"resultBytesPerSec":97.42808374398487,"startupDelaySec":-169.826,"processingDelaySec":77.83,"shutdownDelaySec":66.129,"timeDilation":9.483398748276228,"errors":null,"jobId":null}
2018-11-18T18:08:25.384Z RUNNING Query7
2018-11-18T18:08:26.777Z no activity
2018-11-18T18:08:41.873Z RUNNING Query7
2018-11-18T18:08:43.353Z no activity
2018-11-18T18:08:58.390Z RUNNING Query7
2018-11-18T18:08:59.966Z no activity
2018-11-18T18:09:15.061Z RUNNING Query7
2018-11-18T18:09:16.633Z no activity
2018-11-18T18:09:31.672Z RUNNING Query7
2018-11-18T18:09:33.435Z no activity
2018-11-18T18:09:48.536Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 11,5,main]) completed. Took 6 mins 7.234 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 42s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/23echjarp2j7m
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1074
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1074/display/redirect>
------------------------------------------
[...truncated 265.73 KB...]
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Nov 18, 2018 12:03:46 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackageWithRetry
WARNING: Upload attempt failed, sleeping before retrying staging of package: <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar>
java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-npVr1jgojOCPabH0TiMmrQ.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-npVr1jgojOCPabH0TiMmrQ.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:432)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:287)
at org.apache.beam.runners.dataflow.util.PackageUtil.$closeResource(PackageUtil.java:260)
at org.apache.beam.runners.dataflow.util.PackageUtil.tryStagePackage(PackageUtil.java:260)
at org.apache.beam.runners.dataflow.util.PackageUtil.tryStagePackageWithRetry(PackageUtil.java:203)
at org.apache.beam.runners.dataflow.util.PackageUtil.stagePackageSynchronously(PackageUtil.java:187)
at org.apache.beam.runners.dataflow.util.PackageUtil.lambda$stagePackage$1(PackageUtil.java:171)
at org.apache.beam.sdk.util.MoreFutures.lambda$supplyAsync$0(MoreFutures.java:104)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-npVr1jgojOCPabH0TiMmrQ.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-npVr1jgojOCPabH0TiMmrQ.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 18, 2018 12:03:51 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-npVr1jgojOCPabH0TiMmrQ.jar
Nov 18, 2018 12:03:52 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-npVr1jgojOCPabH0TiMmrQ.jar
Nov 18, 2018 12:03:55 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 28 files newly uploaded
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.ReadBounded as step s1
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource) as step s2
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding NexmarkUtils.GenerateSideInputData/MapElements/Map as step s3
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Monitor as step s4
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Snoop as step s5
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) as step s6
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s7
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike) as step s8
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize as step s9
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize) as step s10
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys as step s11
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) as step s12
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections as step s13
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView as step s14
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) as step s15
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/AsBid as step s16
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles as step s17
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Debug as step s18
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Stamp as step s19
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.Format as step s20
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.DevNull as step s21
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 18, 2018 12:03:56 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <58914 bytes, hash LzEChJGpEJPAziB1dmB6dA> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-LzEChJGpEJPAziB1dmB6dA.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Submitted job: 2018-11-18_04_03_56-5138847349962229062
Nov 18, 2018 12:03:57 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-18_04_03_56-5138847349962229062?project=apache-beam-testing
Nov 18, 2018 12:03:57 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-18_04_03_56-5138847349962229062
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-18T12:03:56.870Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:03:59.951Z: Checking permissions granted to controller Service Account.
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:06.637Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.106Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.281Z: Expanding GroupByKey operations into optimizable parts.
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.329Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.468Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.509Z: Fusing consumer JoinToFiles/JoinToFiles.Monitor into JoinToFiles.ReadBounded
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.554Z: Fusing consumer JoinToFiles/JoinToFiles.Snoop into JoinToFiles/JoinToFiles.Monitor
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.599Z: Fusing consumer JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) into JoinToFiles/JoinToFiles.Snoop
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.640Z: Fusing consumer JoinToFiles/JoinToFiles/justBids/AsBid into JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous)
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.689Z: Fusing consumer JoinToFiles.Format into JoinToFiles/JoinToFiles.Stamp
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.759Z: Fusing consumer JoinToFiles.DevNull into JoinToFiles.Format
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.799Z: Fusing consumer JoinToFiles/JoinToFiles.Stamp into JoinToFiles/JoinToFiles.Debug
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.832Z: Fusing consumer JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles into JoinToFiles/JoinToFiles/justBids/AsBid
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.877Z: Fusing consumer JoinToFiles/JoinToFiles.Debug into JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.922Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Write into JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.947Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Read
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:07.997Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Read
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:08.037Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into NexmarkUtils.GenerateSideInputData/MapElements/Map
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:08.082Z: Fusing consumer NexmarkUtils.GenerateSideInputData/MapElements/Map into NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:08.117Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:08.164Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Write into JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:08.210Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:08.673Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Create
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:08.709Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:08.755Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Create
Nov 18, 2018 12:04:08 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:08.780Z: Starting 4 workers in us-central1-b...
Nov 18, 2018 12:04:10 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:09.014Z: Executing operation NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)+NexmarkUtils.GenerateSideInputData/MapElements/Map+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 18, 2018 12:04:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:04:31.479Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 18, 2018 12:05:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:00.616Z: Workers have started successfully.
Nov 18, 2018 12:05:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:00.898Z: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s).
Nov 18, 2018 12:05:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:00.937Z: Resized worker pool to 2, though goal was 4. This could be a quota issue.
Nov 18, 2018 12:05:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:01.059Z: Workers have started successfully.
Nov 18, 2018 12:05:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:12.478Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 18, 2018 12:05:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:12.516Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 18, 2018 12:05:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:18.664Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 18, 2018 12:05:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:22.751Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 18, 2018 12:05:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:22.840Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Write+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Write
Nov 18, 2018 12:05:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:30.041Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Close
Nov 18, 2018 12:05:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:30.089Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Close
Nov 18, 2018 12:05:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:30.153Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Read+JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey)
Nov 18, 2018 12:05:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:30.194Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Read+JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize)
Nov 18, 2018 12:05:40 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:38.512Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections
Nov 18, 2018 12:05:40 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:38.748Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView
Nov 18, 2018 12:05:40 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:05:38.984Z: Executing operation JoinToFiles.ReadBounded+JoinToFiles/JoinToFiles.Monitor+JoinToFiles/JoinToFiles.Snoop+JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous)+JoinToFiles/JoinToFiles/justBids/AsBid+JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles+JoinToFiles/JoinToFiles.Debug+JoinToFiles/JoinToFiles.Stamp+JoinToFiles.Format+JoinToFiles.DevNull
Nov 18, 2018 12:06:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:06:37.496Z: Cleaning up.
Nov 18, 2018 12:06:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T12:06:37.595Z: Stopping worker pool...
Nov 18, 2018 12:07:58 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-18T12:07:58.050Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-18T12:07:58.086Z RUNNING JoinToFiles
2018-11-18T12:07:59.042Z new perf {"snapshots":null,"runtimeSec":56.761,"numEvents":10000000,"eventsPerSec":176177.30483959056,"eventBytesPerSec":2.2613500414016664E7,"numResults":9200000,"resultsPerSec":162083.12045242332,"resultBytesPerSec":5777413.981430911,"startupDelaySec":-137.918,"processingDelaySec":0.736,"shutdownDelaySec":81.157,"timeDilation":17.616849597434857,"errors":null,"jobId":null}
2018-11-18T12:08:14.087Z RUNNING JoinToFiles
2018-11-18T12:08:15.008Z no activity
2018-11-18T12:08:30.044Z RUNNING JoinToFiles
2018-11-18T12:08:30.920Z no activity
2018-11-18T12:08:45.959Z RUNNING JoinToFiles
2018-11-18T12:08:46.824Z no activity
2018-11-18T12:09:01.862Z DONE JoinToFiles
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 6,5,main]) completed. Took 5 mins 32.12 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 8m 54s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/g7p4mpfkrcbha
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1073
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1073/display/redirect>
------------------------------------------
[...truncated 246.56 KB...]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-fQwy7GdBX2g-kcCwqmZV4g.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-fQwy7GdBX2g-kcCwqmZV4g.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 18, 2018 6:03:57 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-fQwy7GdBX2g-kcCwqmZV4g.jar
Nov 18, 2018 6:04:00 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 18, 2018 6:04:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 18, 2018 6:04:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 18, 2018 6:04:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 18, 2018 6:04:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 18, 2018 6:04:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 18, 2018 6:04:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 18, 2018 6:04:01 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88159 bytes, hash IA3KSrJCz76K6VYzQH_cbg> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-IA3KSrJCz76K6VYzQH_cbg.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 18, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-17_22_04_01-18122786538372324040
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-17_22_04_01-18122786538372324040?project=apache-beam-testing
Nov 18, 2018 6:04:02 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-17_22_04_01-18122786538372324040
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-18T06:04:02.015Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:05.124Z: Checking permissions granted to controller Service Account.
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:11.995Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:12.769Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:13.087Z: Expanding GroupByKey operations into optimizable parts.
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:13.244Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:13.885Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:14.063Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:14.187Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:14.343Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:14.462Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:14.597Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:14.772Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:14.928Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:15.074Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:15.236Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:15.360Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:15.481Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:15.675Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:15.858Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:15.972Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:16.094Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:16.230Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:16.375Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:16.563Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:16.689Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:16.833Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:16.968Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:17.074Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:17.243Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:17.374Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:17.482Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:17.663Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:17.803Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:17.954Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:18.081Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:18.235Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:18.367Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:18.489Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:18.634Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:20.355Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:20.570Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:20.765Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 18, 2018 6:04:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:20.777Z: Starting 4 workers in us-central1-b...
Nov 18, 2018 6:04:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:21.855Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 18, 2018 6:04:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:04:38.068Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 18, 2018 6:05:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:05:05.939Z: Workers have started successfully.
Nov 18, 2018 6:05:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:05:17.965Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 18, 2018 6:05:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:05:37.157Z: Workers have started successfully.
Nov 18, 2018 6:06:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:06:26.399Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 18, 2018 6:06:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:06:26.484Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 18, 2018 6:06:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:06:29.196Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 18, 2018 6:06:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:06:29.302Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 18, 2018 6:06:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:06:30.793Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 18, 2018 6:06:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:06:30.877Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 18, 2018 6:06:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:06:34.073Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 18, 2018 6:06:34 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:06:34.281Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 18, 2018 6:06:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:06:57.720Z: Cleaning up.
Nov 18, 2018 6:06:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T06:06:57.858Z: Stopping worker pool...
Nov 18, 2018 6:08:03 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-18T06:08:03.071Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-18T06:08:03.101Z RUNNING Query7
2018-11-18T06:08:04.465Z new perf {"snapshots":null,"runtimeSec":92.28,"numEvents":10000000,"eventsPerSec":108365.8430862592,"eventBytesPerSec":1.3909459221933246E7,"numResults":100,"resultsPerSec":1.083658430862592,"resultBytesPerSec":109.48201127004768,"startupDelaySec":-157.972,"processingDelaySec":70.76,"shutdownDelaySec":65.692,"timeDilation":10.65669700910273,"errors":null,"jobId":null}
2018-11-18T06:08:19.511Z RUNNING Query7
2018-11-18T06:08:20.602Z no activity
2018-11-18T06:08:35.752Z RUNNING Query7
2018-11-18T06:08:36.978Z no activity
2018-11-18T06:08:52.024Z RUNNING Query7
2018-11-18T06:08:53.210Z no activity
2018-11-18T06:09:08.269Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 5,5,main]) completed. Took 5 mins 34.511 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 1s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/kffueqghcru5w
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1072
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1072/display/redirect>
------------------------------------------
[...truncated 273.62 KB...]
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-0sxwgGaw3fYXWCphtvC51Q.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-0sxwgGaw3fYXWCphtvC51Q.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 18, 2018 12:04:05 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-0sxwgGaw3fYXWCphtvC51Q.jar
Nov 18, 2018 12:04:12 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-0sxwgGaw3fYXWCphtvC51Q.jar
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 18, 2018 12:04:15 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88174 bytes, hash v5JJSZendhWnerfMjxylqw> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-v5JJSZendhWnerfMjxylqw.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 18, 2018 12:04:17 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-17_16_04_16-13198916419471589042
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-17_16_04_16-13198916419471589042?project=apache-beam-testing
Nov 18, 2018 12:04:17 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-17_16_04_16-13198916419471589042
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-18T00:04:16.379Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:19.847Z: Checking permissions granted to controller Service Account.
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:27.325Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:27.808Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:27.917Z: Expanding GroupByKey operations into optimizable parts.
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:27.981Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.215Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.273Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#e9b79e6542892c55
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.324Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.369Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.414Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.461Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.511Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.557Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.607Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.667Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.714Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.774Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.844Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.892Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:28.956Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.003Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.052Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.100Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.146Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.403Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.456Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.501Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.551Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.589Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.652Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.701Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.766Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.812Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.858Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.908Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:29.979Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:30.027Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:30.073Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:30.124Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:30.842Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:30.920Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:31.005Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 18, 2018 12:04:31 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:31.006Z: Starting 4 workers in us-central1-b...
Nov 18, 2018 12:04:32 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:31.400Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 18, 2018 12:04:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:04:43.378Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 18, 2018 12:05:16 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:05:14.195Z: Workers have started successfully.
Nov 18, 2018 12:05:19 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:05:18.650Z: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s).
Nov 18, 2018 12:05:19 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:05:18.709Z: Resized worker pool to 2, though goal was 4. This could be a quota issue.
Nov 18, 2018 12:05:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:05:24.895Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 18, 2018 12:05:47 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:05:45.752Z: Workers have started successfully.
Nov 18, 2018 12:06:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:06:27.712Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 18, 2018 12:06:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:06:27.884Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 18, 2018 12:06:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:06:29.497Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 18, 2018 12:06:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:06:29.627Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 18, 2018 12:06:32 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:06:31.847Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 18, 2018 12:06:32 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:06:32.112Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 18, 2018 12:06:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:06:35.219Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 18, 2018 12:06:36 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:06:35.460Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 18, 2018 12:07:02 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:07:00.989Z: Cleaning up.
Nov 18, 2018 12:07:02 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-18T00:07:01.152Z: Stopping worker pool...
Nov 18, 2018 12:08:17 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-18T00:08:17.415Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-18T00:08:17.456Z RUNNING Query7
2018-11-18T00:08:18.588Z new perf {"snapshots":null,"runtimeSec":92.088,"numEvents":10000000,"eventsPerSec":108591.78177395536,"eventBytesPerSec":1.3938459918773348E7,"numResults":100,"resultsPerSec":1.0859178177395536,"resultBytesPerSec":109.7102771262271,"startupDelaySec":-168.981,"processingDelaySec":69.571,"shutdownDelaySec":76.893,"timeDilation":10.678915819650769,"errors":null,"jobId":null}
2018-11-18T00:08:33.638Z RUNNING Query7
2018-11-18T00:08:34.630Z no activity
2018-11-18T00:08:49.669Z RUNNING Query7
2018-11-18T00:08:50.690Z no activity
2018-11-18T00:09:05.747Z RUNNING Query7
2018-11-18T00:09:06.780Z no activity
2018-11-18T00:09:21.819Z RUNNING Query7
2018-11-18T00:09:22.852Z no activity
2018-11-18T00:09:37.914Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 9,5,main]) completed. Took 5 mins 51.708 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 31s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/go43wfzywac6s
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1071
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1071/display/redirect?page=changes>
Changes:
[gleb] [BEAM-5866] Fix `Row#equals`
------------------------------------------
[...truncated 252.27 KB...]
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-FgNfz2eV-iN6aWbLHtiP0A.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-FgNfz2eV-iN6aWbLHtiP0A.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Nov 17, 2018 7:51:52 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackageWithRetry
WARNING: Upload attempt failed, sleeping before retrying staging of package: <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar>
java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-FgNfz2eV-iN6aWbLHtiP0A.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-FgNfz2eV-iN6aWbLHtiP0A.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:432)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:287)
at org.apache.beam.runners.dataflow.util.PackageUtil.$closeResource(PackageUtil.java:260)
at org.apache.beam.runners.dataflow.util.PackageUtil.tryStagePackage(PackageUtil.java:260)
at org.apache.beam.runners.dataflow.util.PackageUtil.tryStagePackageWithRetry(PackageUtil.java:203)
at org.apache.beam.runners.dataflow.util.PackageUtil.stagePackageSynchronously(PackageUtil.java:187)
at org.apache.beam.runners.dataflow.util.PackageUtil.lambda$stagePackage$1(PackageUtil.java:171)
at org.apache.beam.sdk.util.MoreFutures.lambda$supplyAsync$0(MoreFutures.java:104)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-FgNfz2eV-iN6aWbLHtiP0A.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-FgNfz2eV-iN6aWbLHtiP0A.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 17, 2018 7:51:58 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-FgNfz2eV-iN6aWbLHtiP0A.jar
Nov 17, 2018 7:52:01 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.ReadBounded as step s1
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource) as step s2
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding NexmarkUtils.GenerateSideInputData/MapElements/Map as step s3
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Monitor as step s4
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Snoop as step s5
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) as step s6
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s7
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike) as step s8
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize as step s9
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize) as step s10
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys as step s11
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) as step s12
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections as step s13
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView as step s14
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) as step s15
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/justBids/AsBid as step s16
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles as step s17
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Debug as step s18
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles/JoinToFiles.Stamp as step s19
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.Format as step s20
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding JoinToFiles.DevNull as step s21
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 17, 2018 7:52:02 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <58917 bytes, hash ZgNwwK7r1hvWltTNysKD5Q> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-ZgNwwK7r1hvWltTNysKD5Q.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Submitted job: 2018-11-17_11_52_02-10313966891373507851
Nov 17, 2018 7:52:03 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-17_11_52_02-10313966891373507851?project=apache-beam-testing
Nov 17, 2018 7:52:03 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-17_11_52_02-10313966891373507851
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-17T19:52:02.708Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:05.462Z: Checking permissions granted to controller Service Account.
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:11.788Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.274Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.419Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.444Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.566Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.604Z: Fusing consumer JoinToFiles/JoinToFiles.Monitor into JoinToFiles.ReadBounded
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.630Z: Fusing consumer JoinToFiles/JoinToFiles.Snoop into JoinToFiles/JoinToFiles.Monitor
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.669Z: Fusing consumer JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous) into JoinToFiles/JoinToFiles.Snoop
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.705Z: Fusing consumer JoinToFiles/JoinToFiles/justBids/AsBid into JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous)
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.743Z: Fusing consumer JoinToFiles.Format into JoinToFiles/JoinToFiles.Stamp
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.775Z: Fusing consumer JoinToFiles.DevNull into JoinToFiles.Format
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.812Z: Fusing consumer JoinToFiles/JoinToFiles.Stamp into JoinToFiles/JoinToFiles.Debug
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.848Z: Fusing consumer JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles into JoinToFiles/JoinToFiles/justBids/AsBid
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.885Z: Fusing consumer JoinToFiles/JoinToFiles.Debug into JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.920Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Write into JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.956Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Read
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:12.993Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Read
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:13.029Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into NexmarkUtils.GenerateSideInputData/MapElements/Map
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:13.055Z: Fusing consumer NexmarkUtils.GenerateSideInputData/MapElements/Map into NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:13.087Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike) into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:13.119Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Write into JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:13.154Z: Fusing consumer JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:13.506Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Create
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:13.538Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:13.571Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Create
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:13.587Z: Starting 4 workers in us-central1-b...
Nov 17, 2018 7:52:20 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:13.786Z: Executing operation NexmarkUtils.GenerateSideInputData/GenerateSequence/Read(BoundedCountingSource)+NexmarkUtils.GenerateSideInputData/MapElements/Map+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 17, 2018 7:52:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:23.948Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 17, 2018 7:52:51 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:49.503Z: Workers have started successfully.
Nov 17, 2018 7:52:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:53.219Z: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s).
Nov 17, 2018 7:52:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:53.258Z: Resized worker pool to 2, though goal was 4. This could be a quota issue.
Nov 17, 2018 7:52:53 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:53.567Z: Workers have started successfully.
Nov 17, 2018 7:53:00 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:52:59.068Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 17, 2018 7:53:11 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:53:11.001Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 17, 2018 7:53:11 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:53:11.077Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+JoinToFiles/JoinToFiles/View.AsMap/ParMultiDo(ToIsmRecordForMapLike)+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Write+JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Write
Nov 17, 2018 7:53:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:53:18.117Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Close
Nov 17, 2018 7:53:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:53:18.156Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Close
Nov 17, 2018 7:53:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:53:18.207Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForKeys/Read+JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForKey)
Nov 17, 2018 7:53:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:53:18.272Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/GBKaSVForSize/Read+JoinToFiles/JoinToFiles/View.AsMap/ParDo(ToIsmMetadataRecordForSize)
Nov 17, 2018 7:53:26 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:53:26.713Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/Flatten.PCollections
Nov 17, 2018 7:53:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:53:26.919Z: Executing operation JoinToFiles/JoinToFiles/View.AsMap/CreateDataflowView
Nov 17, 2018 7:53:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:53:27.094Z: Executing operation JoinToFiles.ReadBounded+JoinToFiles/JoinToFiles.Monitor+JoinToFiles/JoinToFiles.Snoop+JoinToFiles/JoinToFiles/justBids/IsBid/ParDo(Anonymous)+JoinToFiles/JoinToFiles/justBids/AsBid+JoinToFiles/JoinToFiles/JoinToFiles.JoinToFiles+JoinToFiles/JoinToFiles.Debug+JoinToFiles/JoinToFiles.Stamp+JoinToFiles.Format+JoinToFiles.DevNull
Nov 17, 2018 7:54:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:54:28.768Z: Cleaning up.
Nov 17, 2018 7:54:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T19:54:28.924Z: Stopping worker pool...
Nov 17, 2018 7:56:03 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
WARNING: No terminal state was returned. State value RUNNING
2018-11-17T19:56:03.794Z Waiting for main pipeline to 'finish'
2018-11-17T19:56:03.824Z RUNNING JoinToFiles
2018-11-17T19:56:04.887Z new perf {"snapshots":null,"runtimeSec":59.19,"numEvents":10000000,"eventsPerSec":168947.45734076703,"eventBytesPerSec":2.1685502568001352E7,"numResults":9200000,"resultsPerSec":155431.66075350565,"resultBytesPerSec":5540324.294644366,"startupDelaySec":-154.429,"processingDelaySec":0.828,"shutdownDelaySec":95.239,"timeDilation":16.89390099679,"errors":null,"jobId":null}
2018-11-17T19:56:19.945Z RUNNING JoinToFiles
2018-11-17T19:56:20.864Z no activity
2018-11-17T19:56:35.907Z RUNNING JoinToFiles
2018-11-17T19:56:36.891Z no activity
2018-11-17T19:56:51.933Z RUNNING JoinToFiles
2018-11-17T19:56:52.964Z no activity
2018-11-17T19:57:08.027Z DONE JoinToFiles
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 6,5,main]) completed. Took 5 mins 32.445 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 23s
76 actionable tasks: 72 executed, 4 from cache
Publishing build scan...
https://gradle.com/s/hvjedbmkbiapa
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1070
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1070/display/redirect>
------------------------------------------
[...truncated 249.29 KB...]
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 17, 2018 6:04:59 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-Gijun_Mjb3-00Hy4AYwBJg.jar
Nov 17, 2018 6:04:59 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-Gijun_Mjb3-00Hy4AYwBJg.jar
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 17, 2018 6:05:02 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 17, 2018 6:05:03 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88175 bytes, hash cnW8_G-y8EWof0i8saNJlQ> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-cnW8_G-y8EWof0i8saNJlQ.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Submitted job: 2018-11-17_10_05_03-11346866341749024139
Nov 17, 2018 6:05:04 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-17_10_05_03-11346866341749024139?project=apache-beam-testing
Nov 17, 2018 6:05:04 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-17_10_05_03-11346866341749024139
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-17T18:05:03.467Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:06.711Z: Checking permissions granted to controller Service Account.
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:13.119Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:13.538Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:13.629Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:13.664Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:13.860Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:13.903Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:13.949Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:13.997Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.044Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.091Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.139Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.189Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.231Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.278Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.320Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.364Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.407Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.451Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.499Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.546Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.590Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.624Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.669Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.714Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.762Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.807Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.847Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.881Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.914Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:14.960Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.006Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.042Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.087Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.136Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.184Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.231Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.272Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.316Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.765Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.814Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.860Z: Starting 4 workers in us-central1-b...
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:15.861Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 17, 2018 6:05:18 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:16.104Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 17, 2018 6:05:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:36.804Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 17, 2018 6:06:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:59.755Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 17, 2018 6:06:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:05:59.794Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 17, 2018 6:06:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:06:05.269Z: Workers have started successfully.
Nov 17, 2018 6:06:06 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:06:05.312Z: Workers have started successfully.
Nov 17, 2018 6:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:06:17.213Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 17, 2018 6:06:19 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:06:17.253Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 17, 2018 6:06:23 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:06:22.697Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 17, 2018 6:07:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:07:27.117Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 17, 2018 6:07:29 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:07:27.215Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 17, 2018 6:07:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:07:29.983Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 17, 2018 6:07:32 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:07:30.084Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 17, 2018 6:07:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:07:32.605Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 17, 2018 6:07:35 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:07:32.701Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 17, 2018 6:07:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:07:35.979Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 17, 2018 6:07:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:07:36.186Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 17, 2018 6:08:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:08:02.438Z: Cleaning up.
Nov 17, 2018 6:08:02 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T18:08:02.603Z: Stopping worker pool...
Nov 17, 2018 6:09:04 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-17T18:09:04.526Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-17T18:09:04.574Z RUNNING Query7
2018-11-17T18:09:05.807Z new perf {"snapshots":null,"runtimeSec":103.15,"numEvents":10000000,"eventsPerSec":96946.19486185166,"eventBytesPerSec":1.2443673262239456E7,"numResults":100,"resultsPerSec":0.9694619486185166,"resultBytesPerSec":97.94474066892874,"startupDelaySec":-165.891,"processingDelaySec":79.787,"shutdownDelaySec":62.741,"timeDilation":9.533688802714492,"errors":null,"jobId":null}
2018-11-17T18:09:20.845Z RUNNING Query7
2018-11-17T18:09:22.067Z no activity
2018-11-17T18:09:37.324Z RUNNING Query7
2018-11-17T18:09:38.528Z no activity
2018-11-17T18:09:53.595Z RUNNING Query7
2018-11-17T18:09:54.800Z no activity
2018-11-17T18:10:09.852Z RUNNING Query7
2018-11-17T18:10:11.327Z no activity
2018-11-17T18:10:26.376Z RUNNING Query7
2018-11-17T18:10:27.685Z no activity
2018-11-17T18:10:42.727Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 5 mins 59.094 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 36s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/rrfa3htrbbk6g
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1069
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1069/display/redirect>
------------------------------------------
[...truncated 241.17 KB...]
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-EO2v1OooyrCHVe_Hq5b2KQ.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-EO2v1OooyrCHVe_Hq5b2KQ.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 17, 2018 12:03:48 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-EO2v1OooyrCHVe_Hq5b2KQ.jar
Nov 17, 2018 12:03:51 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 117 files cached, 28 files newly uploaded
Nov 17, 2018 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 17, 2018 12:03:51 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 17, 2018 12:03:52 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88179 bytes, hash REcoXUgBiXhG79yclr0fxg> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-REcoXUgBiXhG79yclr0fxg.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 17, 2018 12:03:53 PM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-17_04_03_52-16600435183677834497
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-17_04_03_52-16600435183677834497?project=apache-beam-testing
Nov 17, 2018 12:03:53 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-17_04_03_52-16600435183677834497
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-17T12:03:52.676Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:03:55.784Z: Checking permissions granted to controller Service Account.
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:02.363Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.211Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.317Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.364Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.568Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.611Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.648Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.695Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.742Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.788Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.835Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.882Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.927Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:03.975Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.021Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.066Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.105Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.148Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.193Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.234Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.281Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.327Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.373Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.419Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.465Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.502Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.547Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.583Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.629Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.676Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.721Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.760Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.798Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.833Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.880Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.922Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:04.969Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:05.006Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:05.490Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:05.537Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:05.574Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:05.585Z: Starting 4 workers in us-central1-b...
Nov 17, 2018 12:04:09 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:05.888Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 17, 2018 12:04:24 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:24.101Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 17, 2018 12:04:49 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:49.193Z: Workers have started successfully.
Nov 17, 2018 12:04:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:53.331Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 17, 2018 12:04:55 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:53.364Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 17, 2018 12:04:59 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:57.434Z: Workers have started successfully.
Nov 17, 2018 12:04:59 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:04:59.064Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 17, 2018 12:06:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:06:05.834Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 17, 2018 12:06:07 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:06:05.941Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 17, 2018 12:06:10 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:06:07.596Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 17, 2018 12:06:10 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:06:07.680Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 17, 2018 12:06:10 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:06:08.149Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 17, 2018 12:06:10 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:06:08.263Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 17, 2018 12:06:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:06:11.874Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 17, 2018 12:06:13 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:06:12.075Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 17, 2018 12:06:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:06:37.168Z: Cleaning up.
Nov 17, 2018 12:06:37 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T12:06:37.277Z: Stopping worker pool...
Nov 17, 2018 12:07:53 PM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-17T12:07:53.497Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-17T12:07:53.534Z RUNNING Query7
2018-11-17T12:07:54.669Z new perf {"snapshots":null,"runtimeSec":92.967,"numEvents":10000000,"eventsPerSec":107565.0499639657,"eventBytesPerSec":1.380667222777975E7,"numResults":100,"resultsPerSec":1.0756504996396572,"resultBytesPerSec":108.67296997859455,"startupDelaySec":-170.674,"processingDelaySec":70.958,"shutdownDelaySec":77.707,"timeDilation":10.577947013456388,"errors":null,"jobId":null}
2018-11-17T12:08:09.707Z RUNNING Query7
2018-11-17T12:08:10.749Z no activity
2018-11-17T12:08:25.780Z RUNNING Query7
2018-11-17T12:08:26.699Z no activity
2018-11-17T12:08:41.732Z RUNNING Query7
2018-11-17T12:08:42.777Z no activity
2018-11-17T12:08:57.819Z RUNNING Query7
2018-11-17T12:08:58.776Z no activity
2018-11-17T12:09:13.842Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 5,5,main]) completed. Took 5 mins 40.338 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 7s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/h7mejxpzuliyw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1068
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1068/display/redirect?page=changes>
Changes:
[ankurgoenka] Update dataflow container version to beam-master-20181116
------------------------------------------
[...truncated 266.59 KB...]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone
{
"code" : 429,
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-zEIGYVtdaKz0g4LZuf-6uw.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-zEIGYVtdaKz0g4LZuf-6uw.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 17, 2018 6:31:56 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-zEIGYVtdaKz0g4LZuf-6uw.jar
Nov 17, 2018 6:31:59 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 17, 2018 6:32:00 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88187 bytes, hash mDIyWdfFqy4yFAhL4eVhOg> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-mDIyWdfFqy4yFAhL4eVhOg.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 17, 2018 6:32:01 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-16_22_32_00-982867446819691785
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-16_22_32_00-982867446819691785?project=apache-beam-testing
Nov 17, 2018 6:32:01 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-16_22_32_00-982867446819691785
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-17T06:32:01.040Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:04.035Z: Checking permissions granted to controller Service Account.
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:08.769Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.256Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.327Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.354Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.508Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.537Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#8bc528a076ecd5d7
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.570Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.602Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.652Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.683Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.720Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.756Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.794Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.831Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.869Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.908Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.945Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:09.978Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.016Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.050Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.088Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.123Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.160Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.197Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.232Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.272Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.302Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.341Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.379Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.415Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.452Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.501Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.531Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.566Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.604Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.645Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.707Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:10.751Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:11.147Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 17, 2018 6:32:11 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:11.185Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 17, 2018 6:32:13 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:11.219Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 17, 2018 6:32:13 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:11.233Z: Starting 4 workers in us-central1-b...
Nov 17, 2018 6:32:13 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:11.475Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 17, 2018 6:32:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:22.795Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 17, 2018 6:32:49 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:47.919Z: Workers have started successfully.
Nov 17, 2018 6:32:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:51.746Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 17, 2018 6:32:54 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:32:52.829Z: Workers have started successfully.
Nov 17, 2018 6:34:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:34:06.921Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 17, 2018 6:34:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:34:07.050Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 17, 2018 6:34:10 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:34:09.503Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 17, 2018 6:34:10 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:34:09.589Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 17, 2018 6:34:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:34:11.451Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 17, 2018 6:34:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:34:11.531Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 17, 2018 6:34:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:34:14.745Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 17, 2018 6:34:15 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:34:14.909Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 17, 2018 6:34:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:34:39.607Z: Cleaning up.
Nov 17, 2018 6:34:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:34:39.712Z: Stopping worker pool...
Nov 17, 2018 6:36:01 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-17T06:36:01.935Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-17T06:36:01.970Z RUNNING Query7
2018-11-17T06:36:03.066Z new perf {"snapshots":null,"runtimeSec":95.999,"numEvents":10000000,"eventsPerSec":104167.75174741405,"eventBytesPerSec":1.3370606954239108E7,"numResults":100,"resultsPerSec":1.0416775174741404,"resultBytesPerSec":105.24067959041241,"startupDelaySec":-178.678,"processingDelaySec":73.973,"shutdownDelaySec":82.679,"timeDilation":10.243856706840697,"errors":null,"jobId":null}
2018-11-17T06:36:18.103Z RUNNING Query7
2018-11-17T06:36:19.143Z no activity
2018-11-17T06:36:34.180Z RUNNING Query7
2018-11-17T06:36:35.146Z no activity
2018-11-17T06:36:50.188Z RUNNING Query7
2018-11-17T06:36:51.166Z no activity
2018-11-17T06:37:06.211Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 7,5,main]) completed. Took 5 mins 33.443 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 10m 14s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/ruxj5qqntjw5o
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1067
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1067/display/redirect>
------------------------------------------
[...truncated 270.42 KB...]
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-xiPTwA88ZpBl4Dy-Iwz4Jg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-xiPTwA88ZpBl4Dy-Iwz4Jg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 17, 2018 6:04:03 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-xiPTwA88ZpBl4Dy-Iwz4Jg.jar
Nov 17, 2018 6:04:04 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-xiPTwA88ZpBl4Dy-Iwz4Jg.jar
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 17, 2018 6:04:07 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 17, 2018 6:04:08 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88173 bytes, hash 2SDjnlOOY_7BGU9K4Vwa5g> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-2SDjnlOOY_7BGU9K4Vwa5g.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 17, 2018 6:04:09 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-16_22_04_08-2015607738991981482
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-16_22_04_08-2015607738991981482?project=apache-beam-testing
Nov 17, 2018 6:04:09 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-16_22_04_08-2015607738991981482
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-17T06:04:08.463Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:10.896Z: Checking permissions granted to controller Service Account.
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:16.200Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:16.764Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:16.836Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:16.875Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.046Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.077Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.113Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.148Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.185Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.226Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.265Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.298Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.339Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.390Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.427Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.463Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.508Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.563Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.591Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.625Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.659Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.697Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.735Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.755Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.794Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.825Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.855Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.883Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.918Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.955Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:17.993Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.030Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.068Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.105Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.140Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.176Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.214Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.240Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.634Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.660Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.687Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.719Z: Starting 4 workers in us-central1-b...
Nov 17, 2018 6:04:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:18.928Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 17, 2018 6:04:32 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:31.997Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 17, 2018 6:04:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:55.665Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 17, 2018 6:04:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:55.693Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 17, 2018 6:04:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:04:59.275Z: Workers have started successfully.
Nov 17, 2018 6:05:00 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:05:00.431Z: Workers have started successfully.
Nov 17, 2018 6:05:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:05:01.780Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 17, 2018 6:06:16 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:06:15.687Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 17, 2018 6:06:16 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:06:15.777Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 17, 2018 6:06:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:06:17.325Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 17, 2018 6:06:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:06:17.418Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 17, 2018 6:06:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:06:21.927Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 17, 2018 6:06:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:06:22.007Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 17, 2018 6:06:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:06:24.738Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 17, 2018 6:06:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:06:24.944Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 17, 2018 6:06:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:06:50.645Z: Cleaning up.
Nov 17, 2018 6:06:51 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T06:06:50.759Z: Stopping worker pool...
Nov 17, 2018 6:08:09 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-17T06:08:09.305Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-17T06:08:09.344Z RUNNING Query7
2018-11-17T06:08:10.367Z new perf {"snapshots":null,"runtimeSec":97.273,"numEvents":10000000,"eventsPerSec":102803.45008378482,"eventBytesPerSec":1.319548998180379E7,"numResults":100,"resultsPerSec":1.0280345008378482,"resultBytesPerSec":103.8623256196478,"startupDelaySec":-176.19,"processingDelaySec":73.216,"shutdownDelaySec":78.917,"timeDilation":10.109691281239398,"errors":null,"jobId":null}
2018-11-17T06:08:25.404Z RUNNING Query7
2018-11-17T06:08:26.344Z no activity
2018-11-17T06:08:41.414Z RUNNING Query7
2018-11-17T06:08:42.432Z no activity
2018-11-17T06:08:57.480Z RUNNING Query7
2018-11-17T06:08:58.440Z no activity
2018-11-17T06:09:13.481Z RUNNING Query7
2018-11-17T06:09:14.487Z no activity
2018-11-17T06:09:29.535Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':',5,main]) completed. Took 5 mins 52.126 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 23s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/hpctz4dskl4k4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1066
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1066/display/redirect?page=changes>
Changes:
[github] Remove conscrypt as security provider. (#7056)
------------------------------------------
[...truncated 257.18 KB...]
"errors" : [ {
"domain" : "usageLimits",
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-s7Rf9EvVsVAMSp_2uvVnbg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests.",
"reason" : "rateLimitExceeded"
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-s7Rf9EvVsVAMSp_2uvVnbg.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 17, 2018 2:37:39 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-s7Rf9EvVsVAMSp_2uvVnbg.jar
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 17, 2018 2:37:43 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88181 bytes, hash hQxNtwb88nG5eozx87CAVw> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-hQxNtwb88nG5eozx87CAVw.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 17, 2018 2:37:45 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-16_18_37_44-18310074388801088728
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-16_18_37_44-18310074388801088728?project=apache-beam-testing
Nov 17, 2018 2:37:45 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-16_18_37_44-18310074388801088728
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-17T02:37:44.128Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:46.778Z: Checking permissions granted to controller Service Account.
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:50.990Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.431Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.509Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.546Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.704Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.747Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.789Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.823Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.850Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.887Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.925Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:51.963Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.000Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.039Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.077Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.118Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.157Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.196Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.234Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.270Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.312Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.351Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.381Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.421Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.464Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.501Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.533Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.570Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.608Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.636Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.677Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.712Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.752Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.789Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.829Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.866Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.905Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:52.944Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:53.388Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:53.422Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:53.456Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:53.475Z: Starting 4 workers in us-central1-b...
Nov 17, 2018 2:37:55 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:37:53.733Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 17, 2018 2:38:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:38:04.421Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 17, 2018 2:38:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:38:31.890Z: Workers have started successfully.
Nov 17, 2018 2:38:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:38:32.783Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s).
Nov 17, 2018 2:38:33 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:38:32.820Z: Resized worker pool to 1, though goal was 4. This could be a quota issue.
Nov 17, 2018 2:38:39 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:38:38.512Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 17, 2018 2:38:40 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:38:39.625Z: Workers have started successfully.
Nov 17, 2018 2:39:53 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:39:53.118Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 17, 2018 2:39:53 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:39:53.272Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 17, 2018 2:39:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:39:55.638Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 17, 2018 2:39:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:39:55.719Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 17, 2018 2:39:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:39:56.193Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 17, 2018 2:39:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:39:56.282Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 17, 2018 2:40:00 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:40:00.490Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 17, 2018 2:40:02 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:40:00.727Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 17, 2018 2:40:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:40:28.160Z: Cleaning up.
Nov 17, 2018 2:40:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:40:28.275Z: Stopping worker pool...
Nov 17, 2018 2:41:45 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
WARNING: No terminal state was returned. State value RUNNING
2018-11-17T02:41:45.320Z Waiting for main pipeline to 'finish'
2018-11-17T02:41:45.488Z RUNNING Query7
2018-11-17T02:41:47.169Z new perf {"snapshots":null,"runtimeSec":101.381,"numEvents":10000000,"eventsPerSec":98637.81181878262,"eventBytesPerSec":1.2660803276748108E7,"numResults":100,"resultsPerSec":0.9863781181878262,"resultBytesPerSec":99.65378128051607,"startupDelaySec":-179.936,"processingDelaySec":77.076,"shutdownDelaySec":78.555,"timeDilation":9.700042414259082,"errors":null,"jobId":null}
2018-11-17T02:42:02.377Z RUNNING Query7
2018-11-17T02:42:03.576Z no activity
2018-11-17T02:42:18.627Z RUNNING Query7
2018-11-17T02:42:19.685Z no activity
2018-11-17T02:42:34.724Z RUNNING Query7
2018-11-17T02:42:35.762Z no activity
2018-11-17T02:42:50.805Z RUNNING Query7
2018-11-17T02:42:51.906Z no activity
2018-11-17T02:43:06.956Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 7,5,main]) completed. Took 5 mins 49.161 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 31s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/spihrtrz36las
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1065
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1065/display/redirect>
------------------------------------------
[...truncated 247.03 KB...]
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-fetSJcb_GaLoHemH87T0Ng.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 17, 2018 2:27:42 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-fetSJcb_GaLoHemH87T0Ng.jar
Nov 17, 2018 2:27:44 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-fetSJcb_GaLoHemH87T0Ng.jar
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 17, 2018 2:27:47 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88182 bytes, hash 4l2tBvPE5-8IBKNd-Igm0w> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-4l2tBvPE5-8IBKNd-Igm0w.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Submitted job: 2018-11-16_18_27_48-17261874801735851493
Nov 17, 2018 2:27:48 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-16_18_27_48-17261874801735851493?project=apache-beam-testing
Nov 17, 2018 2:27:48 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-16_18_27_48-17261874801735851493
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-17T02:27:48.159Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:50.963Z: Checking permissions granted to controller Service Account.
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:55.446Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:55.836Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:55.943Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:55.980Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.138Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.175Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#6044f02ac45a6f71
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.209Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.246Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.288Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.325Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.359Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.398Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.436Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.473Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.506Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.544Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.577Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.616Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.652Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.687Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.725Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.762Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.785Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.823Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.862Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.894Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.929Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:56.967Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.005Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.038Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.078Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.114Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.150Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.192Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.225Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.257Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.287Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.327Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.745Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.775Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.800Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:57.824Z: Starting 4 workers in us-central1-b...
Nov 17, 2018 2:27:58 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:27:58.020Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 17, 2018 2:28:17 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:28:15.294Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 17, 2018 2:28:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:28:42.014Z: Workers have started successfully.
Nov 17, 2018 2:28:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:28:44.222Z: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s).
Nov 17, 2018 2:28:44 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:28:44.260Z: Resized worker pool to 2, though goal was 4. This could be a quota issue.
Nov 17, 2018 2:29:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:29:02.006Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 17, 2018 2:29:14 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:29:12.191Z: Workers have started successfully.
Nov 17, 2018 2:30:05 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:30:05.102Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 17, 2018 2:30:05 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:30:05.197Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 17, 2018 2:30:09 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:30:07.797Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 17, 2018 2:30:09 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:30:07.873Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 17, 2018 2:30:09 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:30:08.336Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 17, 2018 2:30:09 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:30:08.420Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 17, 2018 2:30:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:30:12.435Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 17, 2018 2:30:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:30:12.629Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 17, 2018 2:30:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:30:37.517Z: Cleaning up.
Nov 17, 2018 2:30:38 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:30:37.611Z: Stopping worker pool...
Nov 17, 2018 2:31:49 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
WARNING: No terminal state was returned. State value RUNNING
2018-11-17T02:31:49.060Z Waiting for main pipeline to 'finish'
2018-11-17T02:31:49.101Z RUNNING Query7
2018-11-17T02:31:50.330Z new perf {"snapshots":null,"runtimeSec":101.185,"numEvents":10000000,"eventsPerSec":98828.8777980926,"eventBytesPerSec":1.268532783515343E7,"numResults":100,"resultsPerSec":0.988288777980926,"resultBytesPerSec":99.84681523941295,"startupDelaySec":-173.1,"processingDelaySec":78.31,"shutdownDelaySec":71.915,"timeDilation":9.718831842664427,"errors":null,"jobId":null}
2018-11-17T02:32:05.374Z RUNNING Query7
2018-11-17T02:32:06.479Z no activity
2018-11-17T02:32:21.525Z RUNNING Query7
2018-11-17T02:32:22.583Z no activity
2018-11-17T02:32:37.618Z RUNNING Query7
2018-11-17T02:32:38.631Z no activity
2018-11-17T02:32:53.682Z RUNNING Query7
2018-11-17T02:32:54.732Z no activity
2018-11-17T02:33:09.776Z RUNNING Query7
2018-11-17T02:33:10.479Z no activity
2018-11-17T02:33:25.479Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 3,5,main]) completed. Took 5 mins 56.086 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 56s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/bsmhoytuzq5c4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1064
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1064/display/redirect?page=changes>
Changes:
[markliu] [BEAM-4850] Remove some dependencies of apitools
[robinyqiu] Redo of buggy PR #6903
------------------------------------------
[...truncated 250.79 KB...]
} ],
"message" : "The total number of changes to the object temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-vwHXRCv8D8_MmajD_doMrA.jar exceeds the rate limit. Please reduce the rate of create, update, and delete requests."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 17, 2018 2:17:34 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-vwHXRCv8D8_MmajD_doMrA.jar
Nov 17, 2018 2:17:35 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-vwHXRCv8D8_MmajD_doMrA.jar
Nov 17, 2018 2:17:39 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 17, 2018 2:17:40 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88175 bytes, hash r1PXMPOvWh3oR-thKywbMw> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-r1PXMPOvWh3oR-thKywbMw.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 17, 2018 2:17:41 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-16_18_17_40-12797514707837819512
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-16_18_17_40-12797514707837819512?project=apache-beam-testing
Nov 17, 2018 2:17:41 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-16_18_17_40-12797514707837819512
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-17T02:17:40.796Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:43.416Z: Checking permissions granted to controller Service Account.
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:49.269Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:49.856Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:49.925Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:49.951Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.114Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.153Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#ab02df29d80e6708
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.191Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.230Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.266Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.304Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.342Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.378Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.414Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.452Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.482Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.519Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.555Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.592Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.635Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.672Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.709Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.741Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.770Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.796Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.829Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.865Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.906Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.946Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:50.985Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.026Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.064Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.097Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.132Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.163Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.201Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.239Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.276Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.314Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.659Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.687Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.723Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.736Z: Starting 4 workers in us-central1-b...
Nov 17, 2018 2:17:52 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:17:51.973Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 17, 2018 2:18:05 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:18:02.778Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 17, 2018 2:18:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:18:43.312Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 17, 2018 2:18:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:18:43.340Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 17, 2018 2:18:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:18:44.002Z: Workers have started successfully.
Nov 17, 2018 2:18:45 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:18:44.048Z: Workers have started successfully.
Nov 17, 2018 2:18:50 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:18:49.376Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 17, 2018 2:19:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:19:56.514Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 17, 2018 2:19:56 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:19:56.591Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 17, 2018 2:19:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:19:58.994Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 17, 2018 2:19:59 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:19:59.068Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 17, 2018 2:20:02 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:20:00.580Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 17, 2018 2:20:02 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:20:00.639Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 17, 2018 2:20:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:20:04.241Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 17, 2018 2:20:04 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:20:04.408Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 17, 2018 2:20:30 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:20:30.235Z: Cleaning up.
Nov 17, 2018 2:20:32 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T02:20:30.374Z: Stopping worker pool...
Nov 17, 2018 2:21:41 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
2018-11-17T02:21:41.762Z Waiting for main pipeline to 'finish'
WARNING: No terminal state was returned. State value RUNNING
2018-11-17T02:21:41.797Z RUNNING Query7
2018-11-17T02:21:42.955Z new perf {"snapshots":null,"runtimeSec":92.755,"numEvents":10000000,"eventsPerSec":107810.89968195785,"eventBytesPerSec":1.3838228634574955E7,"numResults":100,"resultsPerSec":1.0781089968195785,"resultBytesPerSec":108.92135194868202,"startupDelaySec":-164.644,"processingDelaySec":69.209,"shutdownDelaySec":71.889,"timeDilation":10.602123874723734,"errors":null,"jobId":null}
2018-11-17T02:21:58.002Z RUNNING Query7
2018-11-17T02:21:59.076Z no activity
2018-11-17T02:22:14.119Z RUNNING Query7
2018-11-17T02:22:15.282Z no activity
2018-11-17T02:22:30.329Z RUNNING Query7
2018-11-17T02:22:31.886Z no activity
2018-11-17T02:22:46.926Z RUNNING Query7
2018-11-17T02:22:48.440Z no activity
2018-11-17T02:23:03.492Z RUNNING Query7
2018-11-17T02:23:04.752Z no activity
2018-11-17T02:23:19.810Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 6,5,main]) completed. Took 6 mins 2.458 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 10m 8s
76 actionable tasks: 67 executed, 9 from cache
Publishing build scan...
https://gradle.com/s/kwtmlwasi65sw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow
#1063
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/1063/display/redirect?page=changes>
Changes:
[ankurgoenka] Revert "[BEAM-6080] Rollback PR/7049 for failing java post-commits"
[ankurgoenka] [BEAM-6080] Optionally check the sha256 for staged artifacts
------------------------------------------
[...truncated 271.26 KB...]
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:358)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Nov 17, 2018 1:10:09 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-X8R93xgBsunEUvNDlqCPqg.jar
Nov 17, 2018 1:10:09 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Dataflow/ws/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT.jar> to gs://temp-storage-for-perf-tests/nexmark/staging/beam-runners-google-cloud-dataflow-java-legacy-worker-2.9.0-SNAPSHOT-X8R93xgBsunEUvNDlqCPqg.jar
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 116 files cached, 29 files newly uploaded
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.ReadBounded as step s1
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Monitor as step s2
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Snoop as step s3
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/IsBid/ParDo(Anonymous) as step s4
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/justBids/AsBid as step s5
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Window.Into()/Window.Assign as step s6
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/BidToPrice as step s7
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map as step s8
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce as step s9
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey as step s10
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues as step s11
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map as step s12
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map as step s13
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map as step s14
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections as step s15
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey as step s16
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues as step s17
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map as step s18
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) as step s19
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s20
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) as step s21
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView as step s22
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7/Query7.Select as step s23
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Debug as step s24
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7/Query7.Stamp as step s25
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.Format as step s26
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Query7.DevNull as step s27
Nov 17, 2018 1:10:12 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to gs://temp-storage-for-perf-tests/nexmark/staging/
Nov 17, 2018 1:10:13 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <88172 bytes, hash beK4lshrhy1zRV0AjA7aIQ> to gs://temp-storage-for-perf-tests/nexmark/staging/pipeline-beK4lshrhy1zRV0AjA7aIQ.pb
Dataflow SDK version: 2.9.0-SNAPSHOT
Nov 17, 2018 1:10:14 AM org.apache.beam.runners.dataflow.DataflowRunner run
Submitted job: 2018-11-16_17_10_13-8264640809961244975
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-16_17_10_13-8264640809961244975?project=apache-beam-testing
Nov 17, 2018 1:10:14 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2018-11-16_17_10_13-8264640809961244975
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2018-11-17T01:10:13.443Z: The requested max number of workers (4) is ignored as autoscaling is explicitly disabled (autoscalingAlgorithm=NONE).
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:16.311Z: Checking permissions granted to controller Service Account.
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:20.529Z: Worker configuration: n1-standard-1 in us-central1-b.
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.120Z: Expanding CoGroupByKey operations into optimizable parts.
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.247Z: Expanding GroupByKey operations into optimizable parts.
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.297Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.498Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.550Z: Unzipping flatten s15 for input s13.org.apache.beam.sdk.values.PCollection.<init>:402#403d1808e3d3838
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.616Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial, through flatten Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Flatten.PCollections, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.654Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.693Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey) into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.745Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.784Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) into Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.823Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.879Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.922Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:21.967Z: Unzipping flatten s15-u45 for input partial-s16-out1-c43
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.011Z: Fusing unzipped copy of Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify, through flatten s15-u45, into producer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.045Z: Fusing consumer Query7/Query7/BidToPrice into Query7/Query7/Window.Into()/Window.Assign
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.076Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.125Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.170Z: Fusing consumer Query7.Format into Query7/Query7.Stamp
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.204Z: Fusing consumer Query7/Query7.Stamp into Query7/Query7.Debug
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.244Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.277Z: Fusing consumer Query7/Query7/justBids/IsBid/ParDo(Anonymous) into Query7/Query7.Snoop
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.324Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.362Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map into Query7/Query7/BidToPrice
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.401Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.429Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.457Z: Fusing consumer Query7/Query7/justBids/AsBid into Query7/Query7/justBids/IsBid/ParDo(Anonymous)
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.490Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.523Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.554Z: Fusing consumer Query7/Query7.Monitor into Query7.ReadBounded
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.583Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.622Z: Fusing consumer Query7/Query7/Window.Into()/Window.Assign into Query7/Query7/justBids/AsBid
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.666Z: Fusing consumer Query7/Query7.Debug into Query7/Query7/Query7.Select
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.705Z: Fusing consumer Query7/Query7.Snoop into Query7/Query7.Monitor
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.735Z: Fusing consumer Query7.DevNull into Query7.Format
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.778Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:22.814Z: Fusing consumer Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write into Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:23.289Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Create
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:23.329Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Create
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:23.368Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Create
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:23.391Z: Starting 4 workers in us-central1-b...
Nov 17, 2018 1:10:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:23.725Z: Executing operation Query7.ReadBounded+Query7/Query7.Monitor+Query7/Query7.Snoop+Query7/Query7/justBids/IsBid/ParDo(Anonymous)+Query7/Query7/justBids/AsBid+Query7/Query7/Window.Into()/Window.Assign+Query7/Query7/BidToPrice+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/WithKeys/AddKeys/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/AddNonce+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PrepareCold/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Write
Nov 17, 2018 1:10:41 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:10:39.265Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s).
Nov 17, 2018 1:11:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:11:02.873Z: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s).
Nov 17, 2018 1:11:03 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:11:02.911Z: Resized worker pool to 2, though goal was 4. This could be a quota issue.
Nov 17, 2018 1:11:05 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:11:05.364Z: Workers have started successfully.
Nov 17, 2018 1:11:05 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:11:05.400Z: Workers have started successfully.
Nov 17, 2018 1:11:10 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:11:09.013Z: Autoscaling: Raised the number of workers to 3 based on the rate of progress in the currently running step(s).
Nov 17, 2018 1:11:10 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:11:09.050Z: Resized worker pool to 3, though goal was 4. This could be a quota issue.
Nov 17, 2018 1:11:22 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:11:20.668Z: Autoscaling: Raised the number of workers to 4 based on the rate of progress in the currently running step(s).
Nov 17, 2018 1:12:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:12:24.135Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Close
Nov 17, 2018 1:12:25 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:12:24.308Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PreCombineHot/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/StripNonce/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/Window.Remerge/Identity/Map+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Partial+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Reify+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Write
Nov 17, 2018 1:12:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:12:26.290Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Close
Nov 17, 2018 1:12:26 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:12:26.386Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/GroupByKey/Read+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Combine.perKeyWithFanout(MaxLong)/PostCombine/Combine.GroupedValues/Extract+Query7/Query7/Combine.GloballyAsSingletonView/Combine.globally(MaxLong)/Values/Values/Map+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)+Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
Nov 17, 2018 1:12:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:12:27.822Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Close
Nov 17, 2018 1:12:28 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:12:27.903Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read+Query7/Query7/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
Nov 17, 2018 1:12:32 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:12:31.166Z: Executing operation Query7/Query7/Combine.GloballyAsSingletonView/CreateDataflowView
Nov 17, 2018 1:12:32 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:12:31.397Z: Executing operation Query7/Query7/Query7.Select+Query7/Query7.Debug+Query7/Query7.Stamp+Query7.Format+Query7.DevNull
Nov 17, 2018 1:12:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:12:56.882Z: Cleaning up.
Nov 17, 2018 1:12:57 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-11-17T01:12:57.080Z: Stopping worker pool...
Nov 17, 2018 1:14:14 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish
WARNING: No terminal state was returned. State value RUNNING
2018-11-17T01:14:14.382Z Waiting for main pipeline to 'finish'
2018-11-17T01:14:14.449Z RUNNING Query7
2018-11-17T01:14:16.384Z new perf {"snapshots":null,"runtimeSec":97.728,"numEvents":10000000,"eventsPerSec":102324.81990831696,"eventBytesPerSec":1.3134054692616241E7,"numResults":100,"resultsPerSec":1.0232481990831697,"resultBytesPerSec":103.37876555337263,"startupDelaySec":-175.836,"processingDelaySec":74.507,"shutdownDelaySec":78.108,"timeDilation":10.062622789783891,"errors":null,"jobId":null}
2018-11-17T01:14:31.447Z RUNNING Query7
2018-11-17T01:14:32.822Z no activity
2018-11-17T01:14:47.872Z RUNNING Query7
2018-11-17T01:14:49.296Z no activity
2018-11-17T01:15:04.395Z RUNNING Query7
2018-11-17T01:15:05.491Z no activity
2018-11-17T01:15:20.548Z RUNNING Query7
2018-11-17T01:15:22.112Z no activity
2018-11-17T01:15:37.179Z RUNNING Query7
2018-11-17T01:15:38.533Z no activity
2018-11-17T01:15:53.603Z DONE Query7
> Task :beam-sdks-java-nexmark:run FAILED
:beam-sdks-java-nexmark:run (Thread[Task worker for ':' Thread 6,5,main]) completed. Took 6 mins 6.062 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':beam-sdks-java-nexmark:run'.
> Process 'command '/usr/local/asfpackages/java/jdk1.8.0_191/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/4.10.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 9m 48s
76 actionable tasks: 71 executed, 5 from cache
Publishing build scan...
https://gradle.com/s/anid5bvtui5h4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org