You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2021/12/04 12:40:18 UTC

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #1575

See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1575/display/redirect?page=changes>

Changes:

[Robert Bradshaw] Revert "Revert "Allow wildcards for java class lookup transform

[Robert Bradshaw] Move stand-alone expansion service jar into its own project.

[noreply] Merge pull request #16117 from [BEAM-13368][Playground][Bugfix] Fix CI

[noreply] Run python SpannerIO IT with python 3.7 only to avoid overload spanner

[noreply] Merge pull request #15378 from [RFC] Define and document per-key

[noreply] Bump python containers to beam-master-20211202 (#16129)

[noreply] [BEAM-13354, BEAM-13015, BEAM-12802, BEAM-12588] Support prefetch for

[zyichi] Fix failing RecommendationAICatalogItemIT

[Valentyn Tymofieiev] Clarify instructions on how to get contributor list.

[noreply] [BEAM-11936] Fix errorprone UnusedVariable in IO (#16036)


------------------------------------------
[...truncated 3.23 MB...]
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1080)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1133)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:378)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$900(BatchedStreamingWrite.java:71)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:266)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:863)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:814)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1673)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1080)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1133)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:378)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$900(BatchedStreamingWrite.java:71)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:266)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:863)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:814)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1673)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1080)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1133)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:378)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$900(BatchedStreamingWrite.java:71)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:266)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:863)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:814)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1673)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1080)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:1133)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.flushRows(BatchedStreamingWrite.java:378)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.access$900(BatchedStreamingWrite.java:71)
        org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite$BatchAndInsertElements.finishBundle(BatchedStreamingWrite.java:266)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/beam_postrelease_mobile_gaming/tables/leaderboard_DataflowRunner_user/insertAll?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Table apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user",
  "status" : "NOT_FOUND"
}
        com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
        com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
        com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
        com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:863)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl$InsertBatchofRowsCallable.call(BigQueryServicesImpl.java:814)
        org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1673)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:748)
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__

Waiting on bqjob_r43c23e5a6d7ed00b_0000017d85757578_1 ... (0s) Current status: RUNNING
                                                                                      
Waiting on bqjob_r43c23e5a6d7ed00b_0000017d85757578_1 ... (0s) Current status: DONE   
+-----------------------------------+
|             table_id              |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct   |
| leaderboard_DataflowRunner_team   |
| leaderboard_DataflowRunner_user   |
| leaderboard_DirectRunner_team     |
+-----------------------------------+
bq query --batch "SELECT user FROM [apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user] LIMIT 10"

Waiting on bqjob_r403efbe15279d72a_0000017d857581ae_1 ... (0s) Current status: RUNNING
                                                                                      
Waiting on bqjob_r403efbe15279d72a_0000017d857581ae_1 ... (0s) Current status: DONE   
+--------------------------+
|           user           |
+--------------------------+
| user11_AquaWallaby       |
| user4_AliceBlueCaneToad  |
| user12_BananaDingo       |
| user0_AmethystAntechinus |
| user9_BeigePossum        |
| user2_FuchsiaWallaby     |
| user5_BananaDingo        |
| user1_AmberMarmot        |
| user6_AliceBlueCaneToad  |
| user5_ArmyGreenKoala     |
+--------------------------+
Verified AliceBlue
RUNNING_JOB=`gcloud dataflow jobs list | grep leaderboard-validation-1638620784552-657 | grep Running | cut -d' ' -f1`
if [ ! -z "${RUNNING_JOB}" ] 
  then 
    gcloud dataflow jobs cancel ${RUNNING_JOB}
  else 
    echo "Job 'leaderboard-validation-1638620784552-657' is not running."
fi 

Cancelled job [2021-12-04_04_26_38-6842756683132824918]

**************************************
* SUCCESS: LeaderBoard successfully run on DataflowRunner. with Streaming Engine
**************************************

[SUCCESS]

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1h 39m 53s
9 actionable tasks: 8 executed, 1 from cache

Publishing build scan...
https://gradle.com/s/oesod2fhpkfme

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #1577

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1577/display/redirect?page=changes>


---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #1576

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1576/display/redirect>

Changes:


------------------------------------------
[...truncated 2.92 MB...]
INFO: Uploading 232 files from PipelineOptions.filesToStage to staging location to prepare for execution.
Dec 05, 2021 12:42:31 PM org.apache.beam.runners.dataflow.util.PackageUtil stageClasspathElements
INFO: Staging files complete: 232 files cached, 0 files newly uploaded in 0 seconds
Dec 05, 2021 12:42:31 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging portable pipeline proto to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/
Dec 05, 2021 12:42:31 PM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage
INFO: Uploading <274045 bytes, hash e5e2c1a3a347824f2d7fc55024a9685cb848cae4f3327f64a20d1335a310cbc1> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-5eLBo6NHgk8tf8VQJKloXLhIyuTzMn9kog0TNaMQy8E.pb
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PubsubIO.Read/PubsubUnboundedSource as step s1
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PubsubIO.Read/MapElements/Map as step s2
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ParseGameEvent as step s3
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CalculateTeamScores/LeaderboardTeamFixedWindows/Window.Assign as step s4
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CalculateTeamScores/ExtractTeamScore/MapElements/Map as step s5
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey as step s6
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues as step s7
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/ConvertToRow as step s8
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) as step s9
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) as step s10
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites as step s11
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds as step s12
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign as step s13
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey as step s14
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable as step s15
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign as step s16
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map as step s17
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) as step s18
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CalculateUserScores/LeaderboardUserGlobalWindow/Window.Assign as step s19
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CalculateUserScores/ExtractUserScore/MapElements/Map as step s20
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey as step s21
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues as step s22
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/ConvertToRow as step s23
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) as step s24
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) as step s25
Dec 05, 2021 12:42:32 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites as step s26
Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds as step s27
Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign as step s28
Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey as step s29
Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable as step s30
Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign as step s31
Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map as step s32
Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) as step s33
Dec 05, 2021 12:42:33 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Dataflow SDK version: 2.36.0-SNAPSHOT
Dec 05, 2021 12:42:34 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-12-05_04_42_33-13296014152000631895?project=apache-beam-testing
Dec 05, 2021 12:42:34 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Submitted job: 2021-12-05_04_42_33-13296014152000631895
Dec 05, 2021 12:42:34 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel --region=us-central1 2021-12-05_04_42_33-13296014152000631895
Dec 05, 2021 12:42:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
WARNING: 2021-12-05T12:42:36.807Z: Autoscaling is enabled for Dataflow Streaming Engine. Workers will scale between 1 and 100 unless maxNumWorkers is specified.
Dec 05, 2021 12:42:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:36.892Z: Autoscaling is enabled for job 2021-12-05_04_42_33-13296014152000631895. The number of workers will be between 1 and 100.
Dec 05, 2021 12:42:38 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:36.923Z: Autoscaling was automatically enabled for job 2021-12-05_04_42_33-13296014152000631895.
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:42.486Z: Worker configuration: e2-standard-2 in us-central1-b.
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.118Z: Expanding CoGroupByKey operations into optimizable parts.
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.284Z: Expanding SplittableProcessKeyed operations into optimizable parts.
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.312Z: Expanding GroupByKey operations into streaming Read/Write steps
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.476Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.669Z: Fusing adjacent ParDo, Read, Write, and Flatten operations
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.698Z: Fusing consumer CalculateUserScores/LeaderboardUserGlobalWindow/Window.Assign into ParseGameEvent
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.734Z: Fusing consumer CalculateTeamScores/LeaderboardTeamFixedWindows/Window.Assign into ParseGameEvent
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.766Z: Fusing consumer PubsubIO.Read/MapElements/Map into PubsubIO.Read/PubsubUnboundedSource
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.791Z: Fusing consumer ParseGameEvent into PubsubIO.Read/MapElements/Map
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.825Z: Fusing consumer CalculateTeamScores/ExtractTeamScore/MapElements/Map into CalculateTeamScores/LeaderboardTeamFixedWindows/Window.Assign
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.850Z: Fusing consumer CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/WriteStream into CalculateTeamScores/ExtractTeamScore/MapElements/Map
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.879Z: Fusing consumer CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues into CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/GroupByKey/ReadStream
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.912Z: Fusing consumer WriteTeamScoreSums/ConvertToRow into CalculateTeamScores/ExtractTeamScore/Combine.perKey(SumInteger)/Combine.GroupedValues
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.935Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) into WriteTeamScoreSums/ConvertToRow
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:44.976Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) into WriteTeamScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous)
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.011Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables)
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.031Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.061Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.093Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/WriteStream into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.126Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/ReadStream
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.158Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.192Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable
Dec 05, 2021 12:42:45 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.225Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.261Z: Fusing consumer WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) into WriteTeamScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.296Z: Fusing consumer CalculateUserScores/ExtractUserScore/MapElements/Map into CalculateUserScores/LeaderboardUserGlobalWindow/Window.Assign
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.319Z: Fusing consumer CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/WriteStream into CalculateUserScores/ExtractUserScore/MapElements/Map
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.350Z: Fusing consumer CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues into CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/GroupByKey/ReadStream
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.385Z: Fusing consumer WriteUserScoreSums/ConvertToRow into CalculateUserScores/ExtractUserScore/Combine.perKey(SumInteger)/Combine.GroupedValues
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.429Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous) into WriteUserScoreSums/ConvertToRow
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.461Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables) into WriteUserScoreSums/BigQueryIO.Write/PrepareWrite/ParDo(Anonymous)
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.514Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/CreateTables/ParDo(CreateTables)
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.569Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/ShardTableWrites
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.641Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/TagWithUniqueIds
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.685Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/WriteStream into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/Window.Into()/Window.Assign
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.723Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/ReadStream
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.757Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/GroupByKey/MergeBuckets
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.792Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/Reshuffle/ExpandIterable
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.825Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/GlobalWindow/Window.Assign
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.847Z: Fusing consumer WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StreamingWrite/BatchedStreamingWrite.ViaBundleFinalization/ParMultiDo(BatchAndInsertElements) into WriteUserScoreSums/BigQueryIO.Write/StreamingInserts/StreamingWriteTables/StripShardId/Map
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:45.883Z: The pubsub read for: projects/apache-beam-testing/subscriptions/java_mobile_gaming_topic.subscription-12875879596016191134 is configured to compute input data watermarks based on custom timestamp attribute timestamp_ms. Cloud Dataflow has created an additional tracking subscription to do this, which will be cleaned up automatically. For details, see: https://cloud.google.com/dataflow/model/pubsub-io#timestamps-ids
Dec 05, 2021 12:42:46 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:46.103Z: Starting 1 workers in us-central1-b...
Dec 05, 2021 12:42:50 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:42:49.895Z: Pub/Sub resources set up for topic 'projects/apache-beam-testing/topics/java_mobile_gaming_topic'.
Dec 05, 2021 12:43:01 PM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2021-12-05T12:43:00.746Z: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
bq query SELECT table_id FROM beam_postrelease_mobile_gaming.__TABLES_SUMMARY__

Waiting on bqjob_r3b6c30ce43cab354_0000017d8a9ed94d_1 ... (0s) Current status: RUNNING
                                                                                      
Waiting on bqjob_r3b6c30ce43cab354_0000017d8a9ed94d_1 ... (0s) Current status: DONE   
+-----------------------------------+
|             table_id              |
+-----------------------------------+
| hourly_team_score_python_dataflow |
| hourly_team_score_python_direct   |
| leaderboard_DataflowRunner_team   |
| leaderboard_DataflowRunner_user   |
| leaderboard_DirectRunner_team     |
+-----------------------------------+
bq query --batch "SELECT user FROM [apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DataflowRunner_user] LIMIT 10"

Waiting on bqjob_rdf4ed6089ceefab_0000017d8a9ee4e1_1 ... (0s) Current status: RUNNING
                                                                                     
Waiting on bqjob_rdf4ed6089ceefab_0000017d8a9ee4e1_1 ... (0s) Current status: DONE   
+---------------------------+
|           user            |
+---------------------------+
| user8_AndroidGreenEmu     |
| user10_AquaKangaroo       |
| user12_AppleGreenCockatoo |
| user6_BananaEmu           |
| user0_FuchsiaCassowary    |
| user12_AntiqueBrassBilby  |
| user9_AliceBlueKangaroo   |
| user17_BarnRedPossum      |
| user8_BeigeBilby          |
| user14_AliceBluePossum    |
+---------------------------+
Verified AliceBlue
RUNNING_JOB=`gcloud dataflow jobs list | grep leaderboard-validation-1638708139995-422 | grep Running | cut -d' ' -f1`
if [ ! -z "${RUNNING_JOB}" ] 
  then 
    gcloud dataflow jobs cancel ${RUNNING_JOB}
  else 
    echo "Job 'leaderboard-validation-1638708139995-422' is not running."
fi 

Cancelled job [2021-12-05_04_42_33-13296014152000631895]

**************************************
* SUCCESS: LeaderBoard successfully run on DataflowRunner. with Streaming Engine
**************************************

[SUCCESS]

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':runners:direct-java:runMobileGamingJavaDirect'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1h 20m 16s
9 actionable tasks: 8 executed, 1 from cache

Publishing build scan...
https://gradle.com/s/6jwa2ywwluf4k

Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org