You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2020/08/20 19:00:09 UTC
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1825
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1825/display/redirect?page=changes>
Changes:
[david.janicek] [BEAM-10292] LocalFileSystem.matchNewResource method throws
[Kamil Wasilewski] [BEAM-10674] Python CoGBK: Remove unnecessary step. Adjust parameters to
[Kamil Wasilewski] [BEAM-10674] Python CoGBK: Add streaming job
[Kyle Weaver] [BEAM-9118] Increase portable_runner_test timeout.
[Kyle Weaver] [BEAM-10770] Remove DataflowPortabilityApiUnsupported annotation.
[david.janicek] [BEAM-10292] changes after review
------------------------------------------
[...truncated 288.63 KB...]
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWauMS28ACKN/////////////////////////////////////////////4AjT6gN25OdMdm0LGyAdvcGkniDRQhlJiemjSZNMR6j0m0amTTE0MjRoNNA09TQBoAGgBoAAGgAAAAA0ABoGgAAaABoMJoyaNNDQxDR6QZIRpMRkYTSYIUyPUMJptQAAAPUADTQADQAA0D1PUAAAAAADINDQABoADQAAAAA0AAGNTxJ5Sm1BiY1ANAwjJiBpmk9EY1NqDJpppkYTEGTQYhiNAaDQGINGammTTT0AIDAmCYENNMJppkNGjIAwhoNAEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRAETCBMTVPyFP0ImjQfooHqPKPUyaZNMRtQANANNDTRkGjaamgBoAAA0AAADQAAADQAAAAAABkkxdG9mECtHhiqy3UmifEsj4khr16MVDemKgusgpLFtcWvO1Kj9my11xk1JdZWjm8lmfenngXK/AKRiuWzTCIQEKFAgJGmJNpAM08879NJcomMa7t4V6pKbGNCUxCCdkwkAYs7+FgKBPHBGgUB4L4oVLVW+lt3bti5BeWbWeaeAsNMwGEzDx3+DSSPEmcETgWqzSNQ2hqnFY9MNAsuLTpCLBbyM054mXnSHiCQVmEMPfyo8VcGG0mVIqxEg8BTucpUFU6SYTCcE5ClYSmEwJ1k/ITU5QEc/4GOH0PRNuc5yUquJFSbmziSNCuGYOfoJTphF13QWzqyGEJYYCMRKHJRIlIbCZpEpIbG8hBDYcWQo25IZLPc0yG0yMRyYzlCeEzTRCg0CZYsYZ5plJWTKymzrYWBDOajCzlVQdiCRlRf02d3G//0Rfomx+5nem+KeWmD4/d5PeSAlVxcHr/Cjphb/S5SXXaV6D10fLIX8Sp2K628Ixpu2TxsfIKJWZOOOOOPjGIQhCGMYhBD/UJuZVjMwYBGYIzawMENI0JUfoOBz2uuRkckW9jFQLp4mmrmFqyE5smJuhkLHw8ygvA0tUKi6WlRhbSLAJOgOGeDeLb/OjTW5iLxxZiLoKOJZSmcIq0QNtDDLzj1xXDVsRcyRGczFYoKRBNcKKCINyULnC75zZB2UoqO0FduG9vY2OjNurclmk5XoutglOEay4RlomUKICmqxQKZjXLnDg3/QrqJYps20zoyRM5MMFzXNhJHrnpPTtJhtEeisSxjc3qKP8d3JqhfUy1Th2l/qOD1apaDS2xMGAYIkqVAzkJDQ63X7NPcVeRyPMyUp35aa985+KKsuwMzBivurrDO7J4QeSeSdUSS5YilAcAuAbgt5iAwBxpHE1IYaeN52dY9+mT0ekGlBhFBmeoer2ouJxkwiZGyL8k048eFu7jJXVfeq3dlewaabZGyuyb4mhJmmyQSbkOCiJKnoZPUu1Pfo69ltdWp4ZeBY1vzTS8JEFy/EIwZcgkHgYIk4czhSIFXIWLNYSyuu4JQJeeh1MszJqKG4i6UCk1v5VB1CmZQVxRK7gRfjFYpVF0SdpOe6nj1SnfFS64qmFcx5VNoFk5q4qKL6AYXczZziOXTFYklWqgwiyGNXi7ACtYmzjFLNdAGtBrB9BHpOOmiLs1KBaQBkLkhjaZxuPh2ArMJeOoQqEwEAShLySJMDxS333imLAEj46BQNhkG1Ag5lEdXs2jsVnfMMcjJc2TRqfPW9sx3acBWullen1egrKBI2LrAlVX1UQBQIBqhErQmAcKAURmyLiJqklIu5IqNI7USGHSlIehPB2GxsKhNSxS1zDUMqA1V1VsQJkEBeQECaZ0mUHANMgxh5T8RigfJYm7NnDvHQc4XEGLf2KsWJcat8rVQ3dSusqVSii9YUCppsh11ChVTdEm6qjQSSwL/tlthMxEcAnbvqiqeGXXQisMyRrlOlPmBU3C1KcrKXQStiJkXZzQOmaCOVUEwNFgoNCknYsTi8VPHKi0uHqBVTS/H7g2oFu1NcMswWvT1A2athCsWgWqAjBvZCtgkAcCwttRKpy4ALgycxUU7EWhdJX3NghAYey9c3KT9cINohA+BQC+BcygnCNu6omAKUl/hPQ/Un24gCYsaa4VZJ7hLjBQbNZGEvBTz2/LW8fQyFqAl6CdpGgZgNADAtgFFOT7WMAeVA4UBhyc3aKsru/syZkUUGJgPAHH3cccGwFI49q5nIDK7d6rOwnLDQ2zVyyyBGMyWCoz3VZMYDHrc48BayhECGiQQw1NYmIO3ljhSQUKfriEPgWseDV18ZBl6xYKhdMZyNIg0WihpIu+ZsZXLfxleE6y+NxLyklrBoUBrGbdpWU4ggVbFFzwDw28BPOzKyquGqxJsEsrU1JZqtBk1MGUpt3FOfBMWLCRYciVqlriKi3g3iv19rQFY0ZjRTWF4UK4samhOkgiiAMLkKftsExNt3MtgpYSVLVoohzLUxkmoIVu9MU555rQbvls8UsPddwxbRII0TYNDQKTLWq5p2BgJKGnYUQp7rLklnuzi0zMKMqpSLattGTErDc2UGwIpauRlZtlpoFiOiUsENg2a8BSr9rFaVcYI0VKvGwGzOLE201Uu7+v69TLNxENS52HyHOxEUEImCEOOCimiCQEIRCIhRjwaIACqgMEUpSmlwJ8QcKeGEkIAQAOxxxKWljzWMqe7WetPuVXXGNgXeOKcffsJikgfUIoVBk9DurOQlQ8PEwgDBhKVigAAAAAAAAAAFgADQACMWLJhQ0CABBBBBBG8FsSedjsM7OttqiK5Ntt22xJ222ltt111OWRffklTfEXOWTJmSbbbboovlVKiTbbduOUugVs5xEX5JSpmixiy4JG0zCjQaUppbfca08Ca0xMYzVVZrmtAUfZ1b+KdmzhwgpCVhduVusuZOy2odcJcawUVjjcKwVN+Ig5gM80DTVQt0UvdjmxqWyuq4CSzNLY2T8zfCg/AF0yNFMF4YM4OkHGZENQYJikMjlYSCdVdcQrpZbIXAXhlgWDBlkGR7G48MUGoM9MM7OG7xmIvlvGlMJyy7GGjJUZKxPgU2iJLNcCK1oRs5+LUmZ9eoVCjCrVUVmPJkagTAMNOMSQyNUEGSgtuNd9p9uxBET8WKWJ38XdlJlpuwzFYCB6M31AmgECEWL33O1zFb5vs/nWZyMCsCZV4+TBmLk4F2/ctRrrT4OkkaSupJGFG04xSRp1apKVa21hqlJjktuO5wwbVpRYwlUUYTcqimmu0qV+yp2G9aGya6ZimsBHnn8NoqYjxyQYSEAAsgAZ+yYmJNxmbAzepaKlvfN1+cnfj8rJPfNbPrfp9ftQK/urX/mhoqP9fyk/v+a0r9eQ+ChhkBFgNAAQfgjBCACgBgGEgkFYBmBbMC6Ck/tuRAaoQ/+iWqNK3Bq1pba2AzwYchMpuQSiD70fsGSBsGmDYNMGwYvuRAaIOkggUhoBtJDbaB9LAjmWg8s0KsaWt6v/4u5IpwoSFXGJbeA==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-20T18:46:26.582714Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-20_11_46_25-9052485001174500532'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0820150257'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-20T18:46:26.582714Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-20_11_46_25-9052485001174500532]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-20_11_46_25-9052485001174500532
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-20_11_46_25-9052485001174500532?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-20_11_46_25-9052485001174500532 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:31.724Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-b.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:32.796Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:32.830Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:32.896Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:32.931Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.075Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.171Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.203Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.250Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.289Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.314Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.345Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.371Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.406Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.430Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.461Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.587Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.614Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.640Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.675Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.701Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.733Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.784Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.822Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.846Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.882Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:33.911Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.256Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.337Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.369Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.380Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.393Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.405Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-b...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.461Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.461Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.524Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.559Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:34.594Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:46.933Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:58.939Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:46:58.982Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:47:14.936Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:48:34.201Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:48:34.230Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.445Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.517Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.567Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.637Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.666Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.699Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.704Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.714Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.767Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.779Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.818Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:52:17.851Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:54:20.598Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:54:20.669Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:54:20.718Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:54:20.783Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:28.034Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:28.088Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:28.143Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:28.210Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:30.691Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:30.779Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:30.849Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:30.876Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:30.906Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:30.931Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:30.974Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:31Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:31.059Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:33.594Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:33.659Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:33.722Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:33.767Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:33.840Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:33.909Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:36.455Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:36.524Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:36.600Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:36.723Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:57:36.768Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:58:28.896Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:58:28.945Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-20T18:58:28.979Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-20_11_46_25-9052485001174500532 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1597949182860/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 32 files in 0.0785515308380127 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 32 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00000-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00001-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00002-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00003-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00004-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00005-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00006-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00007-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00008-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00009-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00010-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00011-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00012-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00013-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00014-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00015-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00016-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00017-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00018-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00019-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00020-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00021-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00022-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00023-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00024-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00025-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00026-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00027-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00028-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00029-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00030-of-00032
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results-00031-of-00032
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 2034bd0934464697a343b29bb5f5d523 and timestamp: 1597950006.5948365:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 823
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597949182860/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1597949182860/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 32 files in 0.0578765869140625 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 824.250s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 47.052 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 50s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/atkfzsnpil7rm
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal :
beam_PerformanceTests_WordCountIT_Py37 #1852
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1852/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1851
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1851/display/redirect?page=changes>
Changes:
[zyichi] Fix PairWithRestrictionFn process no attribute signature error
[Boyuan Zhang] Handle split when truncate observes windows.
[noreply] [BEAM-10821] Add Python SDK typing improvements to CHANGES.md (#12693)
------------------------------------------
[...truncated 290.18 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWbYSHeQACKN/////////////////////////////////////////////4AjT6gN24rhjZtCxsgI9g0k4g0UIZKZHpo0j1NMQPSbSepo0yMENANNBk9TQBpoA0GgAAA0ABoAaAAAA0AA0AGgAaB6hpiaNMjQaDTaQZIQ0ExGIJiaEaj1DT0mm1AANAaPUaNGmQABkAB6gaMgAAAAAABoAAAAAABoAAAAAAGSNigAAAATAJgARgBMjAACYAJgAEGBMRgEwmTAAAAAAAAATAmjIwIMAAAJoQep/qkGITACNMTTABMEaZMBGmmTAEwBGACYAIwARppiYATAAAjTAAJgAAmmAAIMIaNAwAJMABMAATAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASJE0CEwgjI1T8hT9CJ6jQfooHqZqPUyGIxAADQDTQA0GmanqaAaAAA0NAAAAAAAAAAAAAAABlkv6+7mICpLmhTZbjokgE80AmirVq+LivDJXLrIX1i2uTJuxxwWbLzrjKH11FaSbqeiBtWrCueCEPmS7tmkEQgIUKBASNMSbSAZqaJ3yaS5ZMY13LnnqkpsY0JTEIJ2DDQBizvQWAoE8SEeArhoXwojtU72e3cuWLuG6s2s91MJaahhMRnHQ/4NNI8QaAROBarNM2xtTVOMyagaRbeXHRkWi38Z5zhMwHRnhyQWGIMXfSs8TeGK4mVorxkg7/Td5isK50JhMJwTkKVpKYTAnYT8fNTlARzvgI4nhOgbc5zkpV8WK03NnFkaVkMw87USnSEX38/dOvKYgligIxkoclEiUhsJmmSkhsbykENhxpCjckhktFzTIbTIxnXRnLE8Rnm0Co0iZasgaJqFCwmWFLeqhaEM5uMTOXVR1wJGZGD6eh23B/6w32HsnvaXmkxDy1j5Pt8fuJASy8t/1fh+qQW9kcm2XyL0HroePMRwljsd1t4RjBHslpibnaGlqQwwwwxYIIEEEEEIIIEESJ7xwQiEoCGAKI4AID36fUElJiij9Fwuc11ycjlC7lRVC6SJpq9hcspObJibqZCycTOqLQX9VKi6XioxPPlgFzmzmjmWl7FGtSXmnV4JY1dYoM21Vk4ZVow4G2M2WvOuQKuSKnSM0GYrFYpGEyRRQRBmFC3lNz3QKGTDjwyobBHEiX9/xwvjBUfukXERDAoLYoi4coJnGFSiApXaoFMyLsBz8HH8Rb4tU2a0zxxImcqGG9rvAYvxU5DO9B70Q3ZbzMSlG+/g5fyvh/TQGnlqfkUeymFy7koRnRG+vhfLLioGchIZ/VyWwT2tY3OP5WQle765eYmvvQVt2BmYMV9ixrjuzeEHknkmKJJaZJRQ1ZqUiJRahaDVknbihgSBSVigpCz6zn2tVDm5bcWbm8OU1Wjp++Ywi2K8Uu5ycLGufFjWF4g4xXJNZHXGzjU3xNDlmmzQScUN+iOLTUMjpnamvsdei2OoU77Z/ZQfkl2ASIL2GFRrRcYkHgYEk4YzhSIFCQsUawllddwSgS85DrZbnTUVNxF8oFJrkJVh0qkyosiqV/CjBkFapVl8SdxOe8nk1SnBK11ZVsLJjzKbQLLzd5WVYKgYX81b3pHYUxWpJWKsMQspkWAXWQVzE2b5Sz3UB54GoIEEul4NEhdofhWkIZE5MZXjVGX8GwFdel06hCojCQBKEwJIkwPB6/tuVMUANx4Z9QLTIOCBB1Kw9Pr2nrVvd1HGoyfM5aNS5y7smO3SwILo5SU6fOVk/kbJi/lVKqqIAn8A1Qh1wTAOFAKIz5F5E1QlIv5QrNM7MSGHuCgdEeCtNjYVSaljlrmKMzoDTXTWxAmOQFqAgUUOkzg4BpEGUNVBIZIYCWNu3Zxb91Hey8gx8harBY1xy4KuVTd9F1NSrUVYFiQKlLYdlYoVc3VJuus0kksK/7bdaTMZHCJ3cGsrnilx4RYGdI1ymnPmRUvFtqZmUdRK6ImRfoNA6TQRy6gmBtGCg0qE7VjcYBU5FVXF49sFdKPod0bcC3aouzPQFryCgbNOwhWLQLygJQbyYrYJgHAsLejSp8h+C8M3UU9OxloKZLe9aIEGHsvTMys7CEHAQIfAoBewvZwDlGXZUVRBSrb4TkNKTriQBMmVKcqsk9wNxgK00WRZMAU49vq6kieMhEgNp+boWoaANQDAtgFBNTreLAeVA5T5h0dXaIszvV2hNCKIDIwHgDkr+ROHaCkck1ezrYyy7gK3rRzA0Ns1c0soRkMtoq+fatNjAZclVHerWUIhQ15hFFHrFBB2wscKTCuqCsIhABaxoNPXRjmXjFgqFwynHvkGvaKF9F3rMqVz30pXhusvTdpgKEtYNKoNYz79O2mMIFYxRe8I8V3CT0M6wrshqwSbBLM1NSWergZNTBlE27ymjBMWPERacmWKjXFVV3DwCwa+3qCwaM5opYGAKleWtTQnQgiqAMTkKfs8MxNt3sugowkqNXCiHMuTGSaghXcAxznomtBveY0RSxdx3TFtUgjaGwaWkUM1ismnaGEkoadpVCnvM2WWi7eNTM4qzKiRdXuIy41abu2o2BFGr0ZmbhaiBYzoFLDDYNmvAUWDbxYi1VJVcrbTQAIgoD5REKF45jOZB1c8oUhXiKTI4ilKca3V1rXil2M41kDWtpMa+45LYRH5kkffffL8bzF0KcW3MoxEdllkUtLHlsZH7lRtqBym64x6Fd04pyCCwmLSGBQkiUGVa7uRciKh4eJhAGDCUrFAAAAAAAAAAALAAGgAEYsWTChoEACCCCCCN4bok9DJaaGhddXEWSbbbuuiTuuuLrr776ZpGDBllTBEXuWXLnSbbbbqqwSrlVJttu7JKXPq6c4iMGWUqTRaxZsMja5xVpNKU0tzutaeFNagmMZqqw1zWgKvs6uDjHbs4sQKQlaX7tbzNnTturHZCXHMFFg43StFTBjIOZDRNI1FWLeFHvRzY1LZXT8JJZ2nsbJ+ZvsUH4AuGZ4UgWjCqg6UcZjw8gwjJMZnKgmFVVZcgrJZemLgLRmgWHDmkGV7G68KVG2GekGdjDe5DGYJb9pTCcs2xiqy1mWwT4VLhElnuBFi0o2dHHtpmjZtisUYlcqywyZcryBMAxUpRNFM8oIMtDbcedzsD2vhET8GKWKsF/cnJnpOxUFYCBspva4ohEKEWLz2ux7Cp8v1/zrNUjAqAmdeNmwqC5VAuXzlyNdanD00jTV9CRiRtd8pI1K9UlKxbi01ShkktyO9wwbVxVawlWVYjdqqlLLitYNlTtOA0Nk19JimsBHnauHAVIYccYGEg/ALH4GfpbNuU5+2wNHr1RHqmJmvzU38flZZz5rp1dPfT6z6v7K2H4/NB+v3/aH/UTejfbwHwfiuICKAagAg+hGAD8FADAMJBIKwDMC2YCmCh/1uRAboQ1FDt0aNxFFRN7atBnHh0FGm6BKIPvR+wZIGwaYNg0wbBi+5EBtAdCCBSGgG0kNtoH7mBHNNB5VoVg0tb1P/xdyRThQkLYSHeQ",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-27T06:46:30.932652Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-26_23_46_29-1528657746125815905'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0827050318'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-27T06:46:30.932652Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-26_23_46_29-1528657746125815905]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-26_23_46_29-1528657746125815905
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-26_23_46_29-1528657746125815905?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-26_23_46_29-1528657746125815905 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:34.012Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:34.744Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:34.782Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:34.854Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:34.881Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.028Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.064Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.098Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.122Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.151Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.178Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.214Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.248Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.283Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.311Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.344Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.374Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.411Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.444Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.475Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.502Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.536Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.572Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.619Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.661Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.699Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:35.758Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:36.882Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:37.387Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:37.579Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:37.592Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:37.787Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:37.788Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:38.184Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:38.185Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:38.473Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:38.611Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:38.646Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:46:57.059Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:47:02.190Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 5 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:47:02.223Z: JOB_MESSAGE_DETAILED: Resized **** pool to 5, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:47:07.469Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:48:41.871Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:48:41.906Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.139Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.202Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.223Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.288Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.309Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.337Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.341Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.360Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.392Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.406Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.442Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:52:00.477Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:54:17.008Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:54:17.076Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:54:17.118Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:54:17.190Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:08.338Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:08.446Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:08.497Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:08.559Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:11.519Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:11.579Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:11.646Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:11.681Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:11.700Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:11.745Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:11.762Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:11.801Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:11.890Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:14.660Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:14.866Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:15.018Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:15.160Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:15.372Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:15.721Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:18.597Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:18.680Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:18.768Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:18.891Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:57:18.918Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:58:10.706Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:58:10.761Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T06:58:10.794Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-26_23_46_29-1528657746125815905 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598510787296/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05666375160217285 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: bb552e89aa2d4251a0190f1bc5b7ffcc and timestamp: 1598511585.4203074:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 797
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598510787296/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598510787296/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05598330497741699 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 798.614s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 21.271 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 27s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/a46jfcngcp6lc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1850
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1850/display/redirect?page=changes>
Changes:
[leiyiz] implement query 12
[saavan] Fix author names
[noreply] [BEAM-10701] Add codecov config, fix paths hopefully (#12684)
[noreply] Merge pull request #12674 from [BEAM-8258] basic metric feature for
[noreply] Add skipped tests for Series.dt and Series.str methods (#12615)
[noreply] [BEAM-9547] Add a few more DataFrame operations (#12682)
------------------------------------------
[...truncated 289.76 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWdb+P5QACKN/////////////////////////////////////////////4AjT6gN24rhjbNCxsgE9g0k4g0UKZkpkZo0jEwgybSepk0ZNGQyGjEDT1NAGg0AGgAAAaABoAZAAAAyaGgAAaABoPRGRkNNDIZDGkGSCARkYgmJk1Go9Q02pptQAAAPUBo9T1AAAANDQaZPUAAAAAAADQAAAAAANAAAAAABkjYoAAAAEwCYAEYATIwAAmACYABBgTEYBMJkwAAAAAAAAEwJoyMCDAAACaEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRNAhMIIxNU9qZPQmpo0ep4oGjajTammTTEaAA0A00BpkGjaamgGgAAA0AAADQAAADQAAAAAABipf1sqYgKcmZFJlqjPHAJpYBLFVq1sfFcMdYuMhfWLS49cco0YLFh1xrJ3101aKbyaeB6pUhXNBCHzHe2jQCIQEKFAgJGmJNpAM1NE5yaS5VMY13Xwr1iU2MaEpiEE7BhoAxZywWAoE8SEdArB4L4UUbNK+mtW7de9hvLFnOdTCWmoYTEZx0f+DTSPEmgETgWszTNU2ZrHFZNQNItvLjpCLRb6M854mYDpDxBILDEGLv5WeKvDFcTK0V4yQeApuMxWFc6EwmE4JyFK0lMJgTsJ+QmpygI5/wMcLoeibc5zkpV8OK03NnDkaVkMw8/USnSEX39BdOvKYgligIxkoclEiUhsJmmSkhsbykENhuyFG2JDJaLmmQ2mRjOUGcqTxGebIKjSJlqyBomoULCZYUt6yFoQzm4xM5ZVHYAkZkYOm0O4YP/WfBQ1pm2F+qMQ6sY2S70p30gJdaXE7E7R1AuFH5mYWx7wHjpeaOwYlzhb1dAjFiPZLTE3O0NLUhhhhhiwQQIIIIIQQQIIkT3jghEJQEMAURwAQHv0+oJKTFFH9BwOe11yUjky7sIqhcSJpq9hcspObJibqZCycLOqLwL+pFRcLpUYnXywDMc2UOXdLmOMKmNxmZEtVOUxkrpeHDKtCDLAwMyaDHKHAqhxlTJGZzMViqUjCa0UUEQbgoWsTdt3ChlA10YoegoxIl/f8cLpgpv2EXERDAoLYoi4cQStYVKICldqgUzIuYOFBx3QrqBaps1ZnjyRM5QMN7XfAxfipyHs9B70Qyq3WZahRffwdv8r4f0zx48tS8ifNpoHLtkCM6I3bourLWgGcdHabYyG1T3Vd32W5uXlvAymvhOfmisDkDMwYhHBwsbmvdEHUnUnBEktakooao1KhEosQsBqiTuhQYJApG0WGOtch0YpohjbN4Ifn8jqbGPduG4QKyJ8s02yUFfG2Ti10ZYsCqkWOq2hr72m9JoSppr0EmpDiIiTcUMxrXJxe8Jfy2+vU8UxX3exeeaXBIgtXYxF/LlEg6DDknBmcERAoGCiTVkrsjmCUCXnodbLc6aipuIvlApNcdKsOn0mVFkVSv4EYMgrVKsviTuJz3M8msU40rXWlWwsmPMptAsvN3lZVgqBhfzVvOI5hMVqSVirDELKZFgF14FcxNnFqWe6gOug0xAgk0fIzxlyd+FaQhkTZTI6ag27wa4VmkvHEIVEYSAJQlxJEmB42VDv1MgAN95KBQLDINKBBvKz9btW3s2Dg1fJoyfOlZbb3uP5LPl6QdV+Bwvg/D99RbxPidnvFuN3UgW8B1HBxlgPrAKIz5F5E1QlIv5MrNM7SSGHSlA9CeDtNjYVSaljlrmKiZkBpLpLYgTJIC8gIE87hMwOAaBBkDyoIzHDASxt273FvnUd+LyDHx1qsFjXGLjVcqm76Lq6lWoqwLEgVKWw7KxQq5uqTddZpJJYV/2260mYyOATu42srnil1sIsDOka5TTnzIqXi1aZmUdRK6ImRfoNA6TQRyygmBsmCg0qE7VjcYBU5BVXF49UK6UfR7c2gFqzPemacLXpqgbFKuhWLILqgJAbkpWwSgNCwt2ilZ5j6FoXt4dk4UtPVJebWCABh5L2fGWn4Eg0iAD0FAMAFq9+bIu+NBVcFLTE7PQfVn2ogCd3epNlcSeYC7Pz+wVxRLgU88vbXUR9jIW8CYoPlSMQwAYgGBcQKKcn2cUA6qBsoDDc3uURDub4wJgRRAdzAdAODfyBx9oKRwWr2ciMsu36t6+cuNDbNbNLKEZDLaKtnWpy5AGTWag75ayhEKGtKIoqOrTkHbCxpSUVlQVREIALOCDSygyTLjFgqFsyHJvkGtZKF9F3zNfK5rshXhuMvjepgKEtqGlUG1M+/TtpjCBWMUXvCPFdwE9DOsK7IasEmwSzNTUlnq4GTUwZRNu8powTFjxEWnJFio1w1Vdx+AWDX2lQWDRnNFLAwBUry1qaE6EEVQBichT9thmJtu9l0FGElRq4UQ5lyYyTUEK7fmOc9E2sG65fRFLF3TbsWzSCNkbBpaRQzWKyadoYSShp2lUKe5zZZaLt3aZnFWZUSLq+IjLjVpuLajYEUavRmZxC1ECxnRKWGGwbNeAosG0ixKyMMbJSsyMBs0C1NtNVru3r+uUln4yGpc7D5HnYiKnntVWteOXX695kDzz1Bjz7WyV0R+VI3333y/RdYudTVvZdGIjkkkalpX8xjKPuU3rMDaTjWOwrvGqbBBXTHpDAoRxKDKlZzVGxFQ8PEwgDBhKVigAAAAAAAAAAFgADQACMWLJhQ0CABBBBBBG8N0SehktNDQuuriLJNtt3XRJ3XXF11999M0jBgyypgiL3LLlzpNttt1VYJVyqk223dklLoFdOcRGDLKVJotYs2GRs84q0mlKaW22+1nhTWoJjGaysNc2sBV9nWwcU7d7ixApCVpfuFuc2dO26sdkJcYwUWDjbq0VMGMg5kNE0jUVYtyUe6HNjUt6t5hJLO09jen5m+ywfgC2ZXRQBeGFUB0Y5DJh1BhGOUytpiUVFVVxiqlh2UtBeGaBYcOaQZXsbfwxUaoz0wzswbrIYzBLfNKYTlm2MVWWsy2CfApcIks9wIsWlG90cerM0bNUrFGJXKssMmXK6gTAMVCQSxSuqCDFhtNdczcDuIERPwYpYqQX9uYmag5FOVgIHpDfVhPCIUIr3Pb7LNU/l+v+dXqEYFMEzLwZcKctqAW7rbka61OP00jTV9CRiRs+LUkalesSlYuItNYoZJLbDvcMG1cVWsJVlWI3CqpSy4rWDeqdpv2hsmumYpqwR6N8DSKwMaJQGEg+gK30GftmZmVbfdZmDwyiGXw+MJOfL5/S1T31a59r+33/FAqoVS79fuio/5/qSlpmdO9xgeh+7GgIkBiACD2EX4PoUAMAwkEgqAMwLjhqoKSluaIDNCDpoZmjTtQZs6e5LAZxobh7k24JRB96P2DJA2DTBsGmDYMX3IgNkDoQQKQ0A2khttA+lgRzTQeWaFYNLa+r/+LuSKcKEhrfx/KA",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-27T00:46:20.953237Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-26_17_46_19-14238271852110793962'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0827000956'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-27T00:46:20.953237Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-26_17_46_19-14238271852110793962]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-26_17_46_19-14238271852110793962
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-26_17_46_19-14238271852110793962?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-26_17_46_19-14238271852110793962 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:23.816Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.165Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.188Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.264Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.292Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.374Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.425Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.447Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.478Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.512Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.549Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.583Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.618Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.656Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.690Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.722Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.752Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.776Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.818Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.846Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.881Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.918Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.946Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:25.969Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.002Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.037Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.082Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.234Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.370Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.401Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.427Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.428Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.455Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-a...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.499Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.499Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.561Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.600Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:26.767Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:48.196Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:46:53.767Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:48:30.088Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:48:30.119Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.480Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.546Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.581Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.644Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.668Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.693Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.702Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.741Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.749Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.773Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.811Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:51:48.838Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:54:06.422Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:54:06.512Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:54:06.566Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:54:06.644Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:46.102Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:46.168Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:46.212Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:46.277Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:48.635Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:48.702Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:48.766Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:48.794Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:48.823Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:48.853Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:48.892Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:48.918Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:48.996Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:51.835Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:51.913Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:51.988Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:52.041Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:52.107Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:52.165Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:54.871Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:54.938Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:55.005Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:55.193Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:56:55.244Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:57:55.944Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:57:55.988Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-27T00:57:56.031Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-26_17_46_19-14238271852110793962 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598489177167/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.0621492862701416 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 60efc2d6594849f09ec6854cf672295f and timestamp: 1598489980.6893268:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 803
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598489177167/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598489177167/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.048793792724609375 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 804.065s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 26.546 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 26s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/qld2eamv33sek
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1849
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1849/display/redirect?page=changes>
Changes:
[sychen] Refactor to reuse computation+key
[Luke Cwik] [BEAM-10771] Fix whitespace error causing whitespace lint to fail
[noreply] [BEAM-4980] Upgrade to Netty 4.1.51.Final and Netty_tcnative
[noreply] [BEAM-10670] Make Twister2Runner opt-out for using an SDF powered Read
------------------------------------------
[...truncated 289.74 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWWutV5IACKN/////////////////////////////////////////////4AjT6gN7cXBGzaFjZAJxjUTiDRQhkp6J4jSPU0xHqPSbSepk09I0ZDIDTQNPUAAAAAAAAaABoAAAAAGgaAABoAGgxGhoNNDQyDJ6QZIRkEZGEIyBTyj1DR6mm1AAAA0Bo9T1AAAAABo00AAAAAAADQAAAAAAAAAAAAAkwAEwABMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADJ/qeoUp70UjNR6gAGg0AAaDQA0AAAAAADQAGgAAAAyABoAAADIAAGQAAAAAAAkwAEwABMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABIkTQITCTIyNU/I1T9CJo0Higeo8o9TJpk0xGgANANNAaZNGh6amgAAAANAAAA0AAAA0AAAADQABjphWd1MYFeXLCsy5VokhE80ImjsWLODHfGSyX2RQLFxcmsP1asN3dtfeZRAuurQTdT0QuppiXPDEIDJeXDUCIQEKFAgJGmJNpAM09A5uaS5RMY13Xwj1CU2MaEpiEE65k0AYs7yFgKBPEBGgWQ4L4QVbda/nuXt7avIrq7t5pp4S00zCYjOPHf4NJI8SbIInAtRmkbU2hqHFZNMNEtvLjoiLRb2M850mYDojxBILDEGLv5WeKvDFcTK0V4yQeApucxWFc6EwmE4JyFK0lMJgTsJ+RmpygI5/no4fQdC25znJSr4kVpubOJI0bIZh5+olOkIvv8LdOvKYgligIxkoclEiUhsJmkSkhsbykENhxZCjcEhktBzTIbTIxnJjOUJ4jPNmFRoky1ZA0DTKFhMsKW9ZC0IZzUYmcqqjsASMyMHSbLuGD/1ZvMXXPY3urvGPzm/8TtcbtpASbEt91Pf+qEW8j8jKsY9/D/zvFHXoSZ1u4WkI291gIjQaPS6nWbBa1rW7UpKUpSpSkpJ/r86g5G1QWhCNJavl9vyqRSqfCeh4PO6y5GRyRd2EVQumiaavYXLKTmyYm6mQsnDzqi4GDTiovlpUY2wFgGV5o5Y784twTGrLvLxCxVJTgLrLrqRsXbOqFWgB+4Muc3elSMJOkhoM5WLBSMJqxRQRBtihcxty3MKGUDzRjh0NWNEwsLjRgGGvBYRcZEUKgtiiL44wmeYVKICldqgUzIuXOHBwOgXTy1TZrzPIEiZyYYb2u+BJHqHofNsJZcMKisSxLs7qKQd77MvOU1MtMw67Lzm406peDS2RNGgaIkqVAzjo7PanV69PaU/cY7k4+S7kpL+2Y/E7UnsGZgxFM2dVfGz6g/E/EzRJLSpKKGrNShRqLkLQask7SUSCQLWVqpx1f1fNcqos2G2Fe5fC6Gpj6RvWcOtjfDLvHEiLk8SFxZXSqXhXrHOi1xtrinApoSJptkEnZDfIiQoyGQ0r3RoK9XYtjp1O9KuGtU8cuyCRBiywiNRLikg+i/EnEmcQRAoeIjTWEsqz4CUCXn4dbLc6aipuIvlApNcCVYdOpMqLIqlfwYwZBWqVZfEncTnup5NQpvytdaVbCyY8ym0Cy81eVlWCoGF/M282jl0xWpJWKsMQspkWAXXgVzE2cYpZ7qA1oNcQoJdDzmiQv0QRLSIMjemMrTSNq8C0FZhLp9CFRmIgCUJkSRJgd+U++5UxAA3Hgm1AtMguQIPRVnpdaw9Wpbtpi0ZDlSKNK5i/sGO1RAKnz8l0OlzVZOCNqzcFTKaqIAnABqhFrwmAcQAURnyLyJqhKRfyRWaR2kkMOjKB6I8FabGwqk1LHLWMdUzoDWXWWxAmRQFyAgUUPkzg4ZqEGUOVDIZIoSWNu3eYt66jvxeQY+BarBY1xq36uVTd9F1dSrUVYFiQKlLYdlYoVc3VJuus0UksK/7bdaTMZHBJ3b+srnil1sIsDOkaxTSnzAqXi2tMzKOoldETIv2TQOk0EcqoJgbNgoNGhO1Y3GAVOOVVxePahXSj8brm2AuW6Lwz0Ba9JUDd1rSFYtgtUBKDfTFbBMA8FhbaqUvkQgYhs9CkJ1stBRZP2LRDAxAF6PdJTUOQXCGECFALuGLaFYW+5KiaApKV98zE9GadiAJq1obCnJQENa4Wot1lmJkCmYCCXu4+hkLGBKzfwnnIbgOQDAsoE7MTTqzAPygYTZh5enuMrz5BtybkUYGpgPoHH38ccK0FI49q9nIDLLt8revnLDQ2zUzSyhGQy2is5rp82QBl1Wkd6tZQiJDZmEcdXUqCDtBY8UmFlUNgRiEC3iA1soMiy+YsFQvTKclAQbNsoYEXfs1srnwJSvJvsvzeJgKEtUNGoNUz79K2mMIFYxRe8I8V3BT2WdYV2Q1YJNglmampLPVwMmpgyibd5TQgmLHiItORLFRriKq7hYBYNbbVBYNGc0UsDAFSvLWpoToQRVAGJyFP2+GYm272XQUYSVGrhRDmXJjJNQQrt8Y5z0DVg3fLaApYu6a7FtEgjZmwaOiUM1ismnaGEkoadpVCnus2WWg7eLTM4qzKiRdXt4y41abm2o2BFGr0ZmbdaaBYzoVLDDYNmtAUWDbRYlZGGNmpWZGA2bItTbTVa7t7DrlJZ+MhqXOQ+Q5yIipztRWteZLtdc5kLnOqMdA89LaRIJkkggggMFVrF0KeW7KoxEfllkUtLXlsZV9yu63C9WfeY2Jd08p6GG0mCkUKhJGoMpsv6g9HIkIfwWMj75JJJJJJJJLwktklx48f0tlZAAAAfmIIfQDxp8+Yx2hEeIAADGIIYxjTGNttrkJaccelbiENgvXsgIAAAClOF2VEAABkAThHTEpQhDj0laXRaxZsMjaZxVotKU0txr6s8Ka0xMYzUVhrGrAVfa1MHFO3eYsQKQlaX7lbrNnTturHZCXGsFFg411aKmDGQcwGgaJpqsW6KPdjmxqW8XU8JJZ2lsbw/O32WD8IXpmaKgLhi0g6Echkg1BimSYzPVxMKVWFyCwl22YvAuGaBYcOaQZXsa/hio2oz04zswbvIYzBLetKYTlm2MVWVoruBTSH4QQszooEONhWeUy22Cl1sNBVVMJaHCtXrtQJhmOpKJo5mqCDHiuPNf9mFuMERPwYpYphwr2cmeo/HQVgIHSm/siiIRIRavvb7HMV/l+v+dTpIw64JnXiTYtBepAvcB65GstPhaSRpK+hIxI2nGKSNOvUJSsW3tNQoZJLcDvcMG1cVWsJVlWI3KqpSy4rWDeKdpvmhsmukYprAR5uDC4UoYUcUGEhCALIQGfplpaRePVWG7/KRkp7fdFTHw+Pyrsz8180v+n1+02r+6tl+Zyd/X7/s9P/51QQV0ECJyqoCNAcgAhBBGFEIFADAMJBIKwDMCy32iien7QiA6Qif9FukaB2Dp1QWdaDO/h5HqTeQlEH34/aMkDYNMGwaYNgxfdiA2YOhBApDQDaSG20D6OBHMtB5doVg0tX1n/xdyRThQkGutV5I",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-26T18:46:35.544501Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-26_11_46_34-12250844836443732077'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0826150316'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-26T18:46:35.544501Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-26_11_46_34-12250844836443732077]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-26_11_46_34-12250844836443732077
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-26_11_46_34-12250844836443732077?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-26_11_46_34-12250844836443732077 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:38.094Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:38.731Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:38.911Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:38.984Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.015Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.186Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.231Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.269Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.299Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.337Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.373Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.410Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.434Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.460Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.484Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.517Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.551Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.586Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.611Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.642Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.676Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.712Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.746Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.782Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.817Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.852Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:39.887Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.056Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.113Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.136Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.159Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.162Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.184Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.235Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.235Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.307Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.342Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:40.410Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:46:50.502Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:47:05.132Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 7 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:47:05.157Z: JOB_MESSAGE_DETAILED: Resized **** pool to 7, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:47:10.365Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:48:48.384Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:48:48.422Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.462Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.547Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.597Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.667Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.704Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.714Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.740Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.770Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.800Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.822Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.833Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:52:12.882Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:54:33.153Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:54:33.277Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:54:33.332Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:54:33.405Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:18.242Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:18.326Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:18.368Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:18.439Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:21.869Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:21.951Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:22.019Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:22.054Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:22.097Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:22.097Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:22.162Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:22.197Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:22.231Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:24.803Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:24.864Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:24.942Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:24.991Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:25.045Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:25.110Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:27.853Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:27.946Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:28.016Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:28.169Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:57:28.194Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:58:21.315Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:58:21.358Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T18:58:21.382Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-26_11_46_34-12250844836443732077 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598467591454/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.07944035530090332 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 2086d4fa2a6840408127b0d2b8ca63de and timestamp: 1598468431.784637:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 840
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598467591454/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598467591454/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.062335968017578125 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 840.980s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 14 mins 4.668 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 15m 15s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/ovw2qrhiqxcrw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1848
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1848/display/redirect?page=changes>
Changes:
[noreply] [BEAM-10675] Add Python GBK Load Tests for streaming on Dataflow
------------------------------------------
[...truncated 289.95 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWdDf8yAACKN/////////////////////////////////////////////4AjT6gN2OrhjbNCxsgE9g0k4g0kKZkpkbI0m0TCaPSbSep6j1GmTCMmgzUD1PUAGTTQAAAAAaABoaAAAAAaBoaAAaABoPRGRkNPUNDEMnpBkpiATEYTSYmTVNqPUNMmm1AAAA0Bo9T1AAAANDRoaAAAAAAAANAAAAAAA0AAADQAAZI2KAAAABMAmABGAEyMAAJgAmAAQYExGATCZMAAAAAAAABMCaMjAgwAAAmhB6n+qQYhMAI0xNMAEwRpkwEaaZMATAEYAJgAjABGmmJgBMAACNMAAmAACaYAAgwho0DAAkwAEwABMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABIkQCEwhqMTVPaE9CanqNHqfooHqPKPU2pptTT0RoADQDTQNGQepsKaAAAAA0AAADQAAADQAAAAAABipg05QwgVpOAFVduKeN8TSviWGvXpxkN0x0ltcDyhbVHrbcUT9my42xc7yqzWiG+mnfdoogVM/AHjHf2zUCIQEKFAgJGmJNpAM0c879NJcNMY13XonpkpsY0JTEIJ1/JIAtRyoUA0CeKCOAUh0K4YRWquBNbuXLF/BfWbWcaN8sNEvmAzDxv+DQSPEGcETgWmzQNI15pnJ4tEMpZcWnSEWC3cZp0JMvHSHhyQVmAMHfyo8TcGC0mVIqwkg8BTtchUFU6SYTCcE5ClYSmEwJ1k/ITU5QEcTwMcbj9G25znJSq5EVJubORIy1wy/xKCU6YRddxbZ1YzAEsEBGElDkokSkNhM0CUkNjeMghsNwQo2ZIZLPc0yG0yMJzAzhk8BmmuCgykyxYgzzRKSsmVlNnWQsCGc9GBnMqg7AEjIi902d3C9/6Lv0ZZp1feqiyfGmE5Hd5XeQAmVRcLrz9JUC4EnlppVJv4f+j5ZK/iZNZY+5A+OEeyWmJudoaWpDDDDDFgggQQQQQgggQRInvHBCISgIYAojgAgPfp9QSUmKKP6Df9DqLl5HCLewioFyQmmrmFqxk5smJuhkLFxsyhVis4KhUqpQr1VimGLzi4q4laq2avRONypV8aeenHFh6+NBcJRvDSOKuPiocRxx5rzFnLNUjKncPhpwNnd4NhibruwaGTDHBih1+KFEwcHjxeL9Z6uioSIH2gpbRF04glYsoUQFNVigUzEubONBvuOtyWKbNnM8eSJnMBfua74Fq8ZOR9roffhGUU4vK1Innsdt3l/F+mdunl6vkz5lM+5hkgRfRm9eF5RY4AxpKS0et1+zR3Fc3mT5uUmO+1nPfPfmjg3kGMYGEUsWRD0ueznwnwlh5ISiCSRkxkqgzPaBQDJiRuJRII4thZLBJWjXy9tcIwyndCN5/G6etk33grIZQz8k46ciHkHSVj1cjC3lNsGmn2Rl14RgUTpYyK45ByO4R4lXg7Kyjy8ML/ey2urS8M1XlsL5pxWECCpXiD6+XJI57GDJGJMaHIgUNDszJyT2F6BJBDz0OplmZNRQ3EXSgUmt9KoOn0zKCuKJXb+L2IVilUXRJ2k57aeLTKd6VLrSqYVzHkU2gWPnrioovUAwu52zn0c2mKxJKtVBgFjMSvC68CtYmzcqWa6AOOA1g+gk0bHTxlud6BSQBcLJTI4aBtvhWA1mEvm0IahMBAEkSskCRA8TX8b1LGgDeeOhSCgxBxOIOxRfV7Nq7EHv37IHyvOlpbP3uH5LPl5Q6r8DjfB+H76lbpPktfultdxQgW6B0HBwlYPrAKIzZFxE1SSkXcIqNA7SSGHSlIehPB2GrqqhNSwy1DDEZkBqqqqWgTIIC6gIE87ZMwOEahBkDrT8ZjgfJYW7NbBu3Qc+XEGHfWKsWFcot6rVQ3dSurqVSii8sCBU02Q66hQqpuiTdVRlSSvr/tlthMwkb8nbvaiqeCXWwisMyRqFOhPnBU3C0qcjKXQStiJkXZzQOmaCOZUEwNcwUGWknYsLi8KnlVRaXD0gqppfjNobAC3anvzNOFK0xoGzVsIVC0C40BIDdlKliUBgUFORJV+YxCoLnYPqNlLRu8zOqBCAwgC9fwmG0MQcRCCBCQF8CpdBKY6QcCfgUxNT9BE9Rs5EAS1bUlNaQgIS6QVfXp7eSsFQQDBzqI+piFsAmqH40rQLwGgBgLcBRzzZvbwHxIKaEwdXZ5ZRr1W15LyJkFpgHsDg3cqcCwFI4LVzOWGV27xWdfOaGhtmnkljCMRjsFTnOpS40GTWKB36lFCIENMohhi1Wcg7WVsKSilp+uIQ+Bax4NXJjILurUC0LhkOReINNooXkVgL14qmvSFWSbXgG/S8UktiGWgNiZt2hZThCBVsUXO+PBbv087MrKq4arEmwSyNTUlmq0GTUwZSm3cU58ExYcBFhy5WqWuQqLeBeFe1NhQFY0ZjRTWF4KFcWNTQnSQRRAGByFP21+Ym27mWwUsJKlq0UQ5lqYyTUEK3eGGc882MG35rPFLB3TaMWvSCNcaplylJkrVc07AvklDTsKIU9tkxyz3ZuEzMKMipSLatlGPCrDa2UGqIpauRkZslooFhOjUr8Ng2akBSr2witKuL8a5SrxMBszixNtNVLu3ruuUyzcJDUugh8t0ERFDruqKUrNlWOvdW+667UW68xklhEelSN5554vROLVO0xTu/ItEbkkjaUljzVri92s7afZVbYtyBV8xpj79hMYkD7QjhaC6KW9TZCVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEEbv2xJ52Kwzs622qIrk223bbEnbbaW23XXU5JF69jlTeiLnLHjzJNttt0UXpVSok223bilLiq2c4iL2OUqZosYsl+Rr8woytKU0tntNjO+mtETGM01WahsYCj7One5N2a2DACkJWF21W2yZk7Lah1wlyjBRWONorBU3sJBzgZ5lNFVC2xS9uObGpay3V8kszQ1dY/M32WD8AXDK4KgLow6AdFONyIcQYZjlMrKwlFDVdUYrpZclLAXRkgV+/kkGN6u08KUGkM9MM7MG3xGEvS3bSmE5ZNXBRjqMdYnv6bRElmuBFayxrZ+HSmZ9ekVCjArVUVmLHjcQJhGGpIJYZXGggxYLbHG8y+5iBET8VtKFD+DcmJmqNwzlQCB2Q4FIngECEWLvudlmK3zfZ/OrUEYVYEzKx8uHOWUAXLzLUai0eBoJGgrqSRgRr9ypI0atMlKtbKw0ykxSWzHc4YNq0osYSqKMBtVRTTXaVK9rKdhvGhsmumYppwRtBWw4irDFCXBggGICdiDG9k3Nyzp7bGvf2rJr7/hFT3x+Xzs9B9HNs5/X7fehTfhMr/VFR0n7/tLTf5vTsJEQIoog4MwGgAIMAfBBiEgDAGCAQCYAxgLdgncUtNcTwG50T/oxufTuQbt6e4KAY2EDqPai6hKIPvR+wZIGwaYNg0wbBi+5EBrgdJBApDQDaSG20D6WBHOtB5ZoVY0tj6v/4u5IpwoSGhv+ZAA==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-26T12:46:21.678446Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-26_05_46_20-7280835498653275262'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0826100259'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-26T12:46:21.678446Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-26_05_46_20-7280835498653275262]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-26_05_46_20-7280835498653275262
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-26_05_46_20-7280835498653275262?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-26_05_46_20-7280835498653275262 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:24.917Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.041Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.080Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.154Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.183Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.282Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.328Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.364Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.399Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.427Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.461Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.495Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.519Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.541Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.562Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.601Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.634Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.659Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.691Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.737Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.768Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.803Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.838Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.870Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.905Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.937Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:26.961Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.158Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.228Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.271Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.272Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.294Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.306Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.364Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.364Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.433Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.459Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:27.516Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:38.280Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:54.317Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:46:54.343Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:47:22.821Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:48:28.919Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:48:28.956Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:48.714Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:48.858Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:48.884Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:48.965Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:48.992Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:49.013Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:49.027Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:49.047Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:49.084Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:49.096Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:49.127Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:51:49.154Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:54:16.519Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:54:16.581Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:54:16.632Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:54:16.698Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:57.019Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:57.063Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:57.107Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:57.151Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:59.585Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:59.664Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:59.735Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:59.770Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:59.793Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:59.813Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:59.866Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:59.900Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:56:59.972Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:02.675Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:02.733Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:02.803Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:02.868Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:02.934Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:03.008Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:05.592Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:05.664Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:05.738Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:05.867Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:05.913Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:53.564Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:53.607Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T12:57:53.643Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-26_05_46_20-7280835498653275262 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598445977822/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06259560585021973 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 10cf3a07eb344d69b81ca61b13194d8a and timestamp: 1598446776.5965533:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 798
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598445977822/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598445977822/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06139039993286133 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 799.285s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 21.964 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 21s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/dltmwfyl6pteq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1847
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1847/display/redirect?page=changes>
Changes:
[noreply] [BEAM-9680] Add Aggregation Count lesson to Go SDK katas (#12646)
------------------------------------------
[...truncated 289.89 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWU4tn3EACKN/////////////////////////////////////////////4AjT6gN24rhjNtCxsgE9YUk4g0UIaSNHpo0m1NMQPSbSep6T1MTRkNANPUD1PUAGmjQAaAAABoAGgBkAAAABoaAAyABoMRkaDT1DQaDR6QZIQNEZGmQmJo1Go9Q09J6m1AANADIGj1PUAAAA0NBpk9QAAAAAAANAAAAAAA0aAAAaAABkjYoAAAAEwCYAEYATIwAAmACYABBgTEYBMJkwAAAAAAAAEwJoyMCDAAACaEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRNAhMJMjE1T2hT9CJ6jQfooHqPKPUyaZNMRoADQDTQBkGjaamgAAAANAAAA0AAAA0AAAAAAAYiX1bKmICnJmRSZaozRtFSVoliq1a2NivDHWLrIYFi0uPXnaNFtiw864yaBdNWkm6qTNfnnhXUbCIDHd2jQCIQEKFAgJGmJNpAM09A77NJcqmMa7n4R6pKbGNCUxCCdiySAMWcsFgKBPFhHgKwfC+HFGzSvalq3br3cN1Ys57p4S00zCYjOPG/4NJI8QbIInAtVmkahtDVOMyaYaJbeXHREWi30Z5zpMwHRHhyQWGIMXfis8TeGK4mVorxkg7/TdZisK50JhMJwTkKVpKYTAnYT8hNTlARz3gI4nQdC25znJSr4sVpubOLI0bIZh56olOkIvv5+6deUxBLFARjJQ5KJEpDYTNIlJDY3lIIbDjSFG4JDJaDmmQ2mRjOUGcqTxGebMKjRJlqyBoGmULCZYUt6uFoQzm4xM5ZVHXgkZkYOk2XbsH/q/fIqxTjK8Uhq/N47iSvH7SQEutLfdT2/mjFvJDJzK2Qfw/87wx99EudhutvCMWI9ktMTc7Q0tSGGGGGLBBAgggghBBAgiRPeOCEQlAQwBRHABAe/T6gkpMUUf0HC53WXJSOTLuviqF0sTTV7C5ZSc2TE3UyFk4mdUXwYNVKi6XioxPQFgGY50ocu8XcYYVYri8AszK5ypWLXVdBLByZVo4tDFxSyNSmsVUSiZjKViqUhCa4UUEQbkoWsPeN5ChlA48MQPtoxIl9fcgLhbTgqouIiFqgtiiLZwxK4wqUQFK7VApmRcwcSDg9At6WqbNSZ48kTOUDDe13oMX46cl7fRe/EMqt5mWoUYIMDcfM+L9NAfPMUvKmzaYPMuSBGdIblwXFlxQDOPj9Bq5HXp7Sq7jIcrIy3bmJv1zv2oKu9gzMGK6vX1p8YvqD8T8S9EktOkrVmpNXRmyLcOgNSSduKJBIFJWCsR9lkZRsoijdtsLJzOF0NXIU3er4d0beCbd+JERjvxYtdG1mMUyTfSa42MYnApoSZpsUEnVDfIji0tDJad7pbO93ktjqVfdmYNhWfFNrgkQWrsKjClxiQfRgSTiTOIIgUPENjVEqrb4CVhLz0OtludNRU3EXygUmuDKsOmUmVFkVSv4UYMgrVKsviTuJz3c8mqU4BWusKthZMeZTaBZebvKyrBUDC/mre9o5hMVqSVirDELKZFgF1wFcxNm9Us91AeeBphqCTS8CaMuzQQrSEMiclMjxnG4eHXCs0l06hCojCQDAZiwvgennPx6uDfQB1XqZ2ANQiJRE5yPi+TkvI0Xrbm/r7bve8RqHNWdgxK0oCsc/KdDpSikoMjYL4MqlU1BAFBgGoEUsCYBxABRGfIvImqEpF/JlZpHZyQw6MoHoTwVpr66qTUsctYxUTUQGkuktiBMFAX0BAmmdJqA35oEGQPqbGY4Wksbduxi3zqO+F5Bj4NqsFjXHLgK5VN30XVVKtRVgWJAqUth2VihVzdUm66zRSSwr/tt1pMxkcIndwKyueKXWQiwM6RrFNKfMipeLUpmZR1EroiZF+yaB0mgjllBMDZsFBo0J2rG4wCpx6quLx6gV0o/Gbk2oFqzNdmpMFryKgbFKuhWLILygJAbyUrYJQHAsLeopUeS0C0MXkU1OwloXqXnHQQwMQBef2SziHIUcQwgQrAvQWsYR2i7soKngpaZ9s9E9Fw6kATBhSXaqJQENcYSFZKrYS4FPQDNZSB8jIWkCZn/dQtwyAbgGBawKCdcObYA/Kw7T5h4eXtrZHyCZEyImoYGA+gchfx5w7QUjkGr2ciMsu36t66cuNDbNXNLKEZDLaKtn2rS44GTW5x4C1lCIUNaURRUdZmIO3FjhSUVlNqiINAs48GllBgsvGLBULZkOQgINayUMCLvWbCV1LkhXknWXpu0wFCW2DRqDbGffpW0xhArGKL3hHiu4SeyzrCuyGrBJsEszU1JZ6uBk1MGUTbvKaEExY8RFpyRYqNcVVXcPALBrbWoLBozmilgYAqV5a1NCdCCKoAxOQp+2wzE23ey6CjCSo1cKIcy5MZJqCFdvzHOegbaDecvoCli7luWLaJBGzNc0dEoZrFZNO0MJJQ07SqFPd5sstB28amZxVmVEi6vbxlxq03VtRriKNXozM2600CxnQqWGGwbNaAosG1ixKyMMbNSsyMBs2Ram2mq13T13WqSz8ZDUuch8jzkRFT7+sLWvGLr9g+xr779Bj8DjkldEglSOCCCAwUXmLmU4t/LoxEdkkjUtK/nMZRzlN+y1yk64x6Fd04pxra6Y1IWqEcSgyes7q7kRUPDxMIAwYSlYoAAAAAAAAAABYAA0AAjFiyYUNAgAQQQQQREoNUtJqUAmma15Sn1oiIiNapaNa0NbDDC7KsRRU1uxRF7lly50m2226qsEq5VSbbbuySlz6unOIjBllKk0WsWbDI2mcVaLSlNLcbnbTwprTExjNVWGsbaAq+zq4OMduxixApCVpfulu82dO26sdkJccwUWDjcq0VMGMg5kNA0TTVYt2Ue8HNjUthdRwklnaWvsH5m+xwfgC2ZXhQBfGFODpRx2QDyDCMcplcpiUTqqrjFVLD0pcBfGaBYcOaQZXr7nwpUagz0wzsgbzIYzBLfNKYTlm18VWWsy2CfCpcIks9wIsWjGxoY9SZoWahWKMSuVZYZMuV5Al+YqEglileUEGJDacedzbXsMIiZ1ilidt9bqE1KDsUxWAgfkN7WE0IhQivee72eap/N9n85Oci/pgmovHy4UxcnAt3HLkay0+HpJGkr6EjEjab1SRp16pKVi29pqlDJJbgd7hg2riq1hKsqxG6VVKWXFawbCnab9obJrpGKaoEedghRxUBhxKAwkGgCpoDP0TU1Ju/przJ/mGsx6/ZXZ33e/4WOe+Kxws+Xz+k+p+qld9/xQfn9fyh/v+c0TONECPxWkA2AbgAgzCMIGgVgGAYSCQUgGYFrwD0KH+25EByhE/6Kco0TqHLmitroDOODwPSm8BKIPvR+wZIGwaYNg0wbBi+5EBswdCCBSGgG0kNtoH0cCOaaDyzQrBpbb1f/xdyRThQkE4tn3EA==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-26T06:46:33.644315Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-25_23_46_31-13003599906346747561'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0826050300'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-26T06:46:33.644315Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-25_23_46_31-13003599906346747561]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-25_23_46_31-13003599906346747561
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-25_23_46_31-13003599906346747561?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-25_23_46_31-13003599906346747561 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:36.889Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:37.713Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:37.766Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:37.881Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:37.911Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.023Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.057Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.079Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.114Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.149Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.184Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.217Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.251Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.285Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.319Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.351Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.388Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.421Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.445Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.478Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.524Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.559Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.591Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.633Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.666Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.707Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:38.728Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.015Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.086Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.120Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.121Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.157Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.157Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.207Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.207Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.276Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.310Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:46:39.341Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:47:07.926Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:47:07.957Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:47:11.623Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:47:34.794Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:48:46.343Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:48:46.364Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.265Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.307Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.330Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.379Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.398Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.422Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.438Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.446Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.472Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.506Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.526Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:19.637Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:22.333Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:22.379Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:22.417Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:54:22.460Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:00.604Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:00.682Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:00.733Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:00.802Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:03.239Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:03.305Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:03.371Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:03.407Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:03.452Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:03.457Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:03.520Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:03.552Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:03.621Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:06.199Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:06.258Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:06.320Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:06.374Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:06.446Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:06.518Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:10.247Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:10.328Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:10.433Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:10.544Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:57:10.581Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:58:08.483Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:58:08.529Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T06:58:08.561Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-25_23_46_31-13003599906346747561 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598424389157/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06753802299499512 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: d598f2c371294559b75c5d0507b5d8bf and timestamp: 1598425192.6082213:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 803
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598424389157/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598424389157/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06436467170715332 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 804.067s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 26.935 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 35s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/b23qz45mtajgk
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1846
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1846/display/redirect?page=changes>
Changes:
[rehmanmuradali0] Adding key in Spark
[kmjung] Update BigQuery storage API source to v1
[kmjung] Make stream progress reporting work with v1 API
[kmjung] Clean up some leftovers
[kmjung] Address code review feedback
[noreply] Dataframe set_index is not index preserving (#12676)
------------------------------------------
[...truncated 290.27 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWccO+ZwACKN/////////////////////////////////////////////4AjT6gNtJgjZtCxsgE9g0k4g0UIZKYniNI9TTEeo9JtJ6mTT0mRkNANNA09IAGgAAAAAGgAaAAAAABoGgA0BoAGgxGhoNNDQyDR6QZIIBGRhNJiaJPSPUNMmm1APUAGg0ADQAAAANGmmT1AAAANAAADQAAAAAAAAAANAACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQep/qkGITACNMTTABMEaZMBGmmTAEwBGACYAIwARppiYATAAAjTAAJgAAmmAAIMIaNAwAJMABMAATAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASJEARMIEyT1T8jVP0ImjQfooHqPKPUyaZNMRtQANANNDTRkGmaamgGmgAADQAAANAAAANAAAAAAAGTTF070YgK8mYFZlyrPHAJpYBLFYsU4EV8Y6S4yF9YuLj1tyrVgtWnXGsnfXXVoxvJp4HqKIVzQQh8x3tw1AiEBChQICRpiTaQDNPPPATSXJpjGu8c+9QlNjGhKYhBOw5JAGLOUCwFAnigjoFIeC+GFW3Wv5rl27ZvYby1bzrTwFhpmAwmYdH/g0kjxRnBE4FqM0jam0NQ4nHphoFlxadKRYLdxmnOky86U8SSCswhh8CVHi7gw2kypFWIkHgqdxlKgqnSTCYTgnIUrCUwmBOsn5CanKAjnvBxwuh6JtznOSlVw4qTc2cORoVwzBz1BKdMIuu6C2dWQwhLDARiJQ5KJEpDYTNIlJDY3kIIbDiiFG3JDJZ7mmQ2mRiOSGcmTwmaaIUGgTLFjDPNMpKyZWU2ddCwIZzMYWcoqDsgSMqL+nzu53/+rd5iq96211dot9cX/idnjdpICVYFvpP3fmiFvJDIy7CQfg/c7wx96EqdduNqCNwOVzPO9B0XTdV1zjjjjjrWsYxjGtaxi1/7UOh0DQgmGGDfcpvzdK0zI4foN/zusuQkciW9lFQLqImmrmFqyE5smJuhkLHwsygvAv5EqLhdKjE6+WAZbMnMHgXi63AMS2qxpXZXiWCrgsRWDcU6dXKtFGJdCG+ZEksSp0jNBnKxSUiCayUUEQbioXvB79zRB2coqO1FduG9vY2OkNurclmi5XoutglOEaq4JlomUKICmqxQKZjXLHCg33QrqZYps15nSEiZyQYLmuaCSMmeh8uwmHQYVFYliXV2UUgr32Jmep6ZaZf1mPmNvp1S4GlsiZswzRJUqBnHx+e1Or16e0qO4x3Jx8p25aZ9k195+FegZmDFZZMqs9tXxB9J9JkiSWlSUUNWalEsyMaFoNWSdqKJBIFrK5VI+w6vmuFUUa/bCwcvhdDUyFK3rKHWxngmXfiRFyd5GOY3WqxyvWONFrjaxycAmhJGm1QSdUN8iJGkIZDSvVIgb1di2OnU7su3aVXxTLEJEGDHCI1MuKSD4L8ScSZxBECh4iMNYSyrvYJQJeeh1MszJqKG4i6UCk1vpVB1KmZQVxRK7fxfjFYpVF0SdpOe5nj1CnelS68qmFcx5VNoFk5m4qKL6AYXcxZzaOWTFYklWqgwiyGNXi7ECtYmzi1LNdAHXQa4gQSYeNnjLk78K0hDImymR00Db/BshWaS8cQhURhIAlCYkkSYHelvtuVMQANx351QLTIOiBB5Kt9Lq2PqQu7Z4tGR5UkjTOYu65js0YCqc/JdDpc1WTcjaMm5U6nqiAJuAaoRS4JgHEAFEZsi4iapJSLuRKjSO2Ehh0xSHoTwthsbCoTUsUtYxVTMgNZdZbECY9AXkBAnncJmBwTUIMgeVBGY4YCWJuzZw7t0HNlxBi31irFiXGLeq1UN3UrralUoovWFAqabIddQoVU3RJuqo0EksC/7ZbYTMRG/J272oqnhl18IrDMkaxTpT5cVNwtrTlZS6CVsRMi7OaB0zQRyigmBosFBoUk7FicXip41UWlw9qFVNL6PXOIAuW570zTha9NUDarWUKxbBdUBIDfSlbBKA0LC3aqU3kQYYBq8ilJ10s9R5X1rRDAw/l5/bKTkOQdBDCACgF3DBrCL7hclRMwUpL+6biejOOpAE0aUVfUEn+GtkJU2yyzkxBTb/ArnYfMyFlAl5330DgNgHABgRoE/NTjmzgPqgXzph4eXqLsD3BNibEUWGhgPgHHXcacCwFI45q5nHjK7d4rOxnKjQ2zUyyyBGMyWCpzvUpcYDJrFA75ayhEKGmURRVdVnIO2ljSkopVBYEQgAt44Gtrgx7L5iwVC6ZDkHyDTbKF9F37NgK5sVIV5Jxl+b1LyklqhoUBqmbdpWU4ggVbFFzwDw279POzKyquGqxJsEsrU1JZqtBk1MGUpt3FOfBMWLCRYcgVqlrhqi3gXiv1uIoCsaMxoprC8KFcWNTQnSQRRAGFyFP22CYm27mWwUsJKlq0UQ5lqYyTUEK3eGKc881YN1yueKWHu+uxbRII0TYNDQKTLWq5p2BgJKGnYUQp7nLklnuzikzMKMqpSLattGTErDcWUGwIpauRlZtlpoFiOiUsENg2a0BSr+IitKuMEaKlXjYDZnFibaaqXefX9gplm4iGpc5D4/nIiKHntUWtecLs9e8yB556ox59rZLKI/Kkb7775fqusXOpq3ssjERySSNS0s+YxlX3a71uBtZxrHYV3jVNggspgJDAoRxKDKKXNTbEVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEEbwWxJ52Owzs622qIrk223bbEnbbaW23XXU5ZF9+SVN8Rc5ZMmZJtttuii+VUqJNtt245S6BWznERfklKmaLGLLgkbTMKNBpSmlt9fVngTWmJjGairNY1YCj7OpfxLs2cOEFISsLtwtzlzJ2W1DrhLjGCisca6sFTfiIOXDPNA01ULclL3Q5salsrq+AkszS2Nk/M3wYPwBdMroqAvDCoB0c4zIB1BhGOUytriUUKsLjFhLTspaC8MsCwYMsgyPY1/DlBtRnphnaQ3WMxF8t20phOWXYw0ZKjJWJ7+m0RJZrgRWtCNnPxbWZn17UqFGFWqorMeTI6gTBMVSQSxSuqCDJw3Guue1A7r4RE/FiliiDF3ZiZqjkU5WAgekN/SJ4RChFm+9zss1X+b7P51agjBrgmZeOlwpy2gC7im2o1lp8DSSNJXUkjCjacWpI06tQlKtbaw1CkxyW3Hc4YNq0osYSqKMJuFRTTXaVK/ZU7DeNDZNdOxTWAjzcEHQUwYUcUGEhBgLIMGfomJiSd/TWmz9LRct7PbWZr3/D41+b+S6cXfP6fWdV/ZWx/E9P/n9fyg/v+c0MDdBACeqyAjAHAAIQIRhBBhQAwDCQSCsAzAjb7RxQf20ogOUIn/RTlGhdQ5c0NoWgzv4eB6U3gJRB96P2DJA2DTBsGmDYMX3IgNEHSQQKQ0A2khttA+mgRzDQeWaFWNLV9X/8XckU4UJDHDvmcA==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-26T00:46:37.499554Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-25_17_46_36-5641805480378614175'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0826000648'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-26T00:46:37.499554Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-25_17_46_36-5641805480378614175]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-25_17_46_36-5641805480378614175
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-25_17_46_36-5641805480378614175?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-25_17_46_36-5641805480378614175 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:40.464Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:41.968Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.005Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.137Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.166Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.261Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.306Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.341Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.377Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.400Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.431Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.474Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.509Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.541Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.581Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.620Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.654Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.692Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.726Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.762Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.807Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.838Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.870Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.910Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.944Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:42.977Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.010Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.423Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.488Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.512Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.523Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.532Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.554Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-a...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.604Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.604Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.694Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.728Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:43.810Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:46:51.930Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:47:12.742Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:48:52.515Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:48:52.551Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.041Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.145Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.177Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.252Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.286Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.304Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.312Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.341Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.362Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.367Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.393Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:00.420Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:16.380Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:16.465Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:16.523Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:54:16.585Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:55.063Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:55.128Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:55.181Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:55.240Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:57.433Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:57.499Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:57.552Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:57.580Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:57.599Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:57.627Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:57.661Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:57.700Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:56:57.774Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:00.752Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:00.826Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:00.895Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:00.939Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:01.022Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:01.097Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:03.670Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:03.731Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:03.818Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:03.942Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:57:03.970Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:58:06.182Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:58:06.234Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-26T00:58:06.270Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-25_17_46_36-5641805480378614175 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598402793654/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06614279747009277 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: c61791bb8898492da6f831f0cf8cace8 and timestamp: 1598403589.5030398:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 795
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598402793654/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598402793654/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05433320999145508 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 796.367s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 19.217 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 34s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/sygfgbgyeaalq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1845
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1845/display/redirect>
Changes:
------------------------------------------
[...truncated 290.38 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWW8sZzAACKN/////////////////////////////////////////////4AjT6gN2cudMdm0LGyAdvcGoniDRQhlJiemjSMTEeo9JtJ6mTTI0MjRoNNA09EAaaAaAAAAAABoAAANAA0BoDQAaABoHqGmJo00NDINNpBkhGkxMTBCbSaJPSPUNGmm1AAAA9Ro0aZAAAAAAaNAAAAAAAAAAAAAAAAAAAAABjU8SeVE9ATQ9TJiaaZNMhkYjJ6m0E09TCaYjIZA00GJoZMIaYjJhNGQaGhiGI9JobRGhoMjTQNMIyGgaaGmhoMQ0wEMhB6n+qQYhMAI0xNMAEwRpkwEaaZMATAEYAJgAjABGmmJgBMAACNMAAmAACaYAAgwho0DAAkwAEwABMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABIkQBAIEyNU/I1T9CJ6jQeKB6maj1NqaYjEAANANNA0ZDI8ppoAAAA0NAAAAAAAAAAAAAAABlkxtrczCBLJvwsMuxURvieZ8TQ2bNrChwDHaLrIKyxdXHk3Yon7lxrrjKKy5VaCb6eh96mmBc78ArGO/umuEQgIUKBASNMSbSAZqaB3yaS5ZMY13PwT1SU2MaEpiEE61ioAxZ3cLAUCeHCNAtB4L4MRVWMGe7evW7+C+uVZpqYCw1DAYTEeM/waaR4czwicC1WaZtTaGqcZk1A0Sy4tOkIsFvozjniZedIeGJBWYQw99KjxFwYbSZUirGSDv1O5zFQVTpJhMJwTkKVhKYTAnWT8fNTlAR0Hf44vg+ibc5zkpVdDFSbmzoZGjXDMHQUEp0wi67iWzqymEJYYCMZKHJRIlIbCZpkpIbG8pBDYcaQo3BIZLQc0yG0yMZ14ZyxPCZxpBQaJMsWQNA1CkrJlZTZ1cLAhnNxhZy6oOuhIzIv+nn9uv/9ESETY/ezvDjFPbTAyfb5HcSAl1xcDq/D9Uot9I5NqukXwPnQ8eZvglzsV0twRjjdpLzPIm0vMTHHHHHHxjEIQhDGMQghwUBuVwxjOYBEYIyGh9kNG2JEXR8HntdcnI5Qt5UVAuliaauYWrKTmyYm6GQsnFxUKsVnDUKlVKFfVYphz/m1xF4GtVbCvji0aU+nKiNmzSVcxqXDilWfCOoIcwxSlNzCpkiM5mKxSUgCZIooIg2lQqym37gFDJhxoyoefihRMLC4wYJflrWUXCRA+oLYoi8coJnGFCiApqsUCmZF2E4kHH8VbwsU2a8zxxImcqGC5rvISR6p6Tz7P2N4wyKxLFODioozvnamKCqJlqXLsL/QdX1SpaDS2pMGAYIkqVAzzGY0Gs12yT29Wksfy8hLd31zEzN/ihrLwDMwYr7s7QzyyekHsnsnZEkuAIpQHAKnugXUxOMAcaR4NRmGnjZduy24ceL2XUaQCmlMzdL0Kfh71bn0LS3RfimHOThY9z41yX3at3pXrmkjsDZR6b8mhKGmyQScEOAiOM7oZHTvDu/yF5LZ6lTvtYJjW/JMLwkQXL8KjWC45IPQwBJw5nCkQKuQsWawlldeQSgS85DqZZimoobiLpQKTXISqDplMygriiV3Bi/IKxSqLok7Sc91PJqlPAKl1hVMK5jzKbQLLzdxUUX0Awu5qznEdiTFYklWqgwiymRXi62CtYmzeKWc6ANaDKH0Emh5zRGXaK0C0gDIXJjI00jaPAthXXJfOoQqEwEAShLySJMDwev7ySmKAG68M+oG4yDegQdSiOn17R1qzvGGNRk+ZKI1HnLeyY7dMArXRykr0+crKCI2LtBFU6oqIAoIA1QiVoTAOFAKIzpFxE1SSkXcoVGmdnJDD3BSHRnP2GzsqhNSxy1zDEZ0BsLsLYgTIIC8gIFFDpM4OGa5BkDyn4zHA+Sxt2b3DvnQc4XEGPkLFWLGuOXAVqobupXVVKpRResKBU02Q66hQqpuiTdVRopJYF/2y2wmYyOCTt4FRVPDLjwisMUjXKdOfMipuFtaczKXQStiJkXZ7QOmaCOXUEwNJgoNGknYsbi8VPIqi0uHtQqppfi9g2wF2qi/M9AWvSVA3LFtCsVAtUBIDgTFbBMA4FhbYkqXJgQuDJ1FPTsRaGmy/vbhCAw+l6ZqWnq4QbxCB+CgF6C5lBuUddVRMAUs1+E7Dys84EATFjS3KqpPsJcIOsM1kaS8FOvr+tcR9DIWoBrPzlE0DMBoAYEYBQzc82jQHtQOU+YdHV4irK8wDMmZFFBiYD0Bw7uROFYCkcNq5nJDK7d+rOuHMDQ2zVzSyhGQy2CtZrp82OBkyVI7xayhECG1MIYYtSoIOzFjhSYWlP2RCHwKseDY1cZBmAxYKheMhyNYg2qihrIvBZlSufGSFeK6zBN+l5SS1g0aA1jOu07KcYQKtii54B4beCnn4qyquGqxJsEszU1JZytBk1MGUpt3FOhBMWPCRYcmVqlroVRbwrxX6+2oCsaMTRTWF4UK4samhOkgiiAMLkKfs8ExNt3MtgpYSVLVoohzLUxkmoIVu/Mc56BrQbvmNAUsPcthi2iQRpGyaOiUmatVzTsDASUNOwohT3WbLLQdnGpmIozKlItq28ZcasNzZQbIilq5GZm3WogWM6JSwQ2DZrwFKv20VpVxgjSUq8jAbM8sTbTVS7p6zrNMs7GQ1LnYfJc7ERQQjiEIcYFCmIJOIQiARCjGpIQAKqgUSlKU0qgfEFWPDCTagEADskkalpb8pjIvcleqfcsOuMbAu+cU4+/bTCSB9QjhUGU2ndQchKh4eJhAGDCUrFAAAAAAAAAAALAAGgAEYsWTChoEACCCCCCN4LYk8/JYZ+fbbVEVybbbttiTtttLbbrrqc0i+/LKm+IucsuXFJtttuii+VUqJNtt25JS4itnOIi/LKVM0WMWbBI2mIo0WlKaW42NaeBNagmMZqqs1zWgKPs6t/GOze4cIKQlYXblbrNinZbUOuEuOYKKxxsKwVN+Mg5kNA0TUVQt0UvdjmxqW9XUcBJYtPZ3p+ZvscH4AvGZorgvDEpB0I47IhqDEMcxmclEwpVZXGLKXGzFwF4ZoFgwZpBlezseEKDajPSDOyBu8hjL5b5pTCcs2zhoytErwKaQ6+CFmdFAh42VZ5TJUwUvVBoKoi+Wh4WJZWoEwzDXkE0MzVBBloLrjXfYfbrIRE/Bilil/G3pyZ67sNBWAgekODaFEAgQi3ge12HXy/L9f86nSRhygmdePmxKC5SBexjlqNdanC00jTV1JIwo2m8UkalWqSlWtvYapSZJLcDucMG1aUWMJVFGE3KopprtKlfvVOw37Q2TX0mKawEeegA3iojDjjgwkIEBZAgz9LVry3P22Bm+eqK9UzNV+bnPj8rJO/NbPLfp9ftPq/urX/mgof1+/7Rf6jbUj/dw/D8wyAiwGgAIP4RgxAhQAwDCQSCsAzAjL/TRRf62ogNkIejiWyNI4Cjo21sbgZ4EOgpE3QJRB96P2DJA2DTBsGmDYMX3IgNIHSQQKQ0A2khttA/cwI5poPKtCrGlrep/+LuSKcKEg3ljOYA=",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-25T18:46:41.047593Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-25_11_46_39-7253857048984676632'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0825150324'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-25T18:46:41.047593Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-25_11_46_39-7253857048984676632]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-25_11_46_39-7253857048984676632
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-25_11_46_39-7253857048984676632?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-25_11_46_39-7253857048984676632 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:44.004Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:44.868Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:44.904Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:44.991Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.027Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.126Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.164Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.197Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.233Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.265Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.308Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.334Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.372Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.412Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.455Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.491Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.532Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.555Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.586Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.621Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.655Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.702Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.738Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.789Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.819Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.855Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:45.890Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.115Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.215Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.253Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.264Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.276Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.308Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-a...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.381Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.383Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.446Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.478Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:46:46.585Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:47:13.504Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:47:18.438Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:48:48.437Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:48:48.461Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:06.402Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:06.504Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:06.591Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:06.792Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:06.880Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:06.932Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:07.020Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:07.058Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:07.116Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:07.138Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:07.216Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:52:07.277Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:55:14.077Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:55:14.142Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:55:14.212Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:55:14.278Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:05.153Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:05.245Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:05.307Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:05.360Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:39.035Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:39.154Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:39.246Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:39.294Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:39.322Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:39.448Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:39.469Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:39.538Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:39.614Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:42.648Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:42.739Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:42.796Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:42.863Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:42.955Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:43.024Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:45.807Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:45.900Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:46.044Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:46.137Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:58:46.170Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:59:42.735Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:59:42.872Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T18:59:42.942Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-25_11_46_39-7253857048984676632 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598381197363/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.07443714141845703 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: d5e15eac891f40d0a5956e66ac2fc273 and timestamp: 1598382081.2802799:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 883
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598381197363/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598381197363/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.0652778148651123 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 884.497s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 14 mins 47.238 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 15m 55s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/vr5mid2qh3kv4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1844
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1844/display/redirect>
Changes:
------------------------------------------
[...truncated 291.25 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWSc/MscACKN/////////////////////////////////////////////4AjT6gN26rd042bQsbIKT1Y1E4g0kKaekmJ6aNJhMRk9JtJ6mTTBM0jJpo9TQNPRAGQGjQaAAABoAGhoAADQNAyBoaDQDIAGgwmAjTIyMQyekGSENNEZMEI0CT0j1DRppkDQAGTTQ0aNNAAAAAANA0AAANAAADIADRoAAAAAAAAAAGJPZKEbQCYEYaAEwEwRgIwCYBGjAAEwAamRhDRoMAAAEyGg0BpoyYJiYATACMAAE0MTIGhLf6n6pFNtFD1PSaaMTQMQaNMmAIyGhpk0aZGmIAxGTBMmAhpoZBoaaBkYJhDEMTBGTTQaaMCNMjQwmgYmQZNA00AJMABMAATAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASJE0CE0aTE2k1T8plP0JHqNB4oHqPKPU0aYT0RkADQDTQBk0aNimgAAAANAAAA0AAAA0AAAADQABlUw9reDGBNLlhYZeV6ZHCidwnjs2bWDHfmS0X2RQrF4uTW369d13dwPvMphXMrGm6opc2qqJdDohCZL28NYIhAQoUCAkaYk2kAzLoHf5pLhJjGu6+FeqSmxjQlMQgnX8igDFnfAsBQJ4wJABaDQvLivcsYFF5fX1u9iuru5nmXCWmUwmIzDx3+DTSPEmyCJwLVZpmobM1Tjc3KGiW3lx0RFot5Gec6TMB0R4gkFhiDF4ArPFXhiuJlaK8ZIPA02+QrCudCYTCcE5ClaSmEwJ2E/IzU5QEc/z0cPidC25znJSr4sVpubOLI0bIZh5+olOkIvv6C6decYgligIxkoclEiUhsJmmSkhsbziCGw6cQo2pIZLQc0yG0yMZygzhE8RnmkFRoky1ZoaBlKFhMsKW9aC0IZzcYmcqqjsQSMiMHSbLuGD/0Hi1tzoFeIZYVwXRsnMdLyIATScpLvUlbYi5UfoZxPHuYc+z646ME0aEir8D4kWGnJ6ho+94accccccpCEAAAAEIQAGG3yIqgaCJ4hBTEAJCE+iFWWFxT9DwOd11yMjki7kxVC6aJpq9hcs4nNkxN1MhZvDzKi0GHHFRfMBUY4ISwDLZg8IeCaYOFtYc12PqxJWp1kqgRvQZZTXJqOrGoUUuHZHETyPCs9ZykaCd4+EnA2d3g1OY3XdgoZMPQDKhrq8aJg4PIjALpobKLjIicoLYoi9OUE7zBUogKV2qBTM1cucODf8RbotU2bWZ5AkTOUDDe13wAN8CEjjUOUWCQIxSAmRLSsrR3i4zdxGKhZcwtuFjkEyYCXiEAUYpIkISGFCEGNHR2u3G63yPDtkjnurn5nyzc/Q0f5rHxtBjGBhBNbXANypvOcCcCazyQkGQwo+kMWsLyQcEfLddOmgqPPajo+XyGYzeSexyWOoGP73NZznOUv+vt410N6p9nk1kUzykUowz9FpN2u2W9MqwiLqidKmRVHIMh0keJRsO0Eg2tjtGYkuBtkvnnHlS/eyfUBAgnUZM97Lokc3jHkjAmMsIgT+shjMJMNvbgSUIbyHWy3MmoqbiL5QKTW/lWHT6TKiyKpX8CMGaK1SrL4k7ic9xPN1Sm+K11tVsLJjyKbQLO5u8rKsFQML+at76jl0xWpJWKsMQs4zVgF18FcxNm6Us91BBADMHIJdK0GmQv0wxLSIMjenMsBqG25y2FZlLp9CFRmIgElJQSBIgeibquOlzAAkfTUpQxGIMZxBpKDl/BeO++clxzZ8p1pU9y7CbxGExZALd2tF25eWSE8kZS1vJWm1MBAE8gGYAtTBEAywAjoz5F5E1QlIv5IrNM7SSGHRlA9EeDtNjYVSaljlrqOuaEBsLsLYgTFIC1AQKaXyaAcI1iDKGqdIZInCWNu3d4t46jvxeQY9/arBY1xy3yuVTd9F1hSrUVYFiQKlLYdlYoVc3VJuus0UksK/7bdaTMZHAJ3b6srnil1wIsDMka5TTnzIqXi1KZGUdRK6ImRfsmgdJoI5VQTA0mCg0aE7VjcYBU49VXF49QK6UfjdsbQF5cpvTRSFr1BQN3YtoVi4DAoCUG/nK2CcB4LC4K6YPUeAnCppFmRhC17VNUDEH0GDoXxopmnfyDGH0OoSgRYTqnpmicKwE4gpmcpKaB7lOyEASlTY2a1oOj7gHp7VsMQSgFTOjsmZRUGIXoCcqaWuXBWAuAMBDgVlHTr4gBwShmqTBoaW2Furc7qyVkUKFJgG8Bbfx5wbQUjsDV7OQGWXb1W9hOWGhtmrklnBGaZ1otLPcfPiAZdZqGbWsoREhtTiOOvq1JB2sseKTi0p1kRhwFzEg2MmMUy/YsFQvjKcXCQbVwoYUXgM14row0pXkX2YBvUjKEtYNGoNYz79O2mMIFYxRe8I8V3AT2WZYV2Q1YJNglkampLPVwMmpgyibd5TQgmLHiItORLFRriqq7g4BYNfaVBYNGY0UsDAFSvLWpoToQRVAGJyFP2+GYm272XQUYSVGrhRDmXJjJNQQrt6Y5z0DWg3PLaApYu6bZi2aQRpGwaOiUMliawpCBWCypCkIeUWYU8y6khz8oJA9OXyQ6DDKmlMI29tRsCKNXoyM4xZUCxnQqWGGwbNeAosG0ixKyMMaSlZmsBs2Ram2mq13b1/XaSz8ZDUuch8hzkRFVjcgta86Xb7BrHNa2sxsLz0ttEhnSSGGGEw14GLpU8tvAoxEfllkUtLflsZX9yZtxz1h95kES7p5TznW0wUicoSRqDKrT+qvR0GGIyBx2DTTQAAAAAAAEAAQgBEgQIzUIPgiIiIkrOUtKbEIppc6BSmrREREc5S0c5wc6KKJ+dYjjmW/GoxCQSJMICIiIiI8eRDAg8CIiIjsKBuEo400wwxEkAPmpWAtYsmGRs8wq0WlKaW122tPCmsomMZqqw1zWgKvtauDjXbu8WIFIStL9utxkzJ23VjshLjmCiwcbZWipgxkHMhoGiZVWLcFHuRzY1LdrqmEkszT2N2fnb5iD8IdMM8ArAtGMqB0k4jFiBBjDJOZ3phOKlWVyCyl3BOXgWieBYcOSQZz2Nt4YqNQZ6cZ2YNzmmMwS3jSmE5ZNjFVnViZoKae+4ELNCKBDTZVolUtxgqbcEAKq5cYA0WJpoECYRjrSieOeBQQZWK8egf9l0GUCIn4MUsVOw99QTRWfjpKwEDZTgWhTEIkIt3/t5rMzfL9f86vURhTAmheJnxlJeqAvsM85GusvB00jTV9CRiRs90pIy16pKVi4y01ShmyW1He4YNq4qtYSrKsRt1VSllxWsG7U7TetDZNdIxTrBG1buGMOQyolgYIB4AYXgGN8p2dlWeeuKtzm4WboaKCo6X6fW6U32TU6ao+/4qUlUkUfqrrK39/2ur/8vsHbDh1FXAHCGAXAAg7A96DwEoBgDBAIBIAYwEPjmoV1ffTwF50D/lq8+wZAvX2F8YgY0aGgTyLQCUQffj9oyQNg0wbBpg2DF92IDSB0IIFIaAbSQ22gfRwI5poPLtCsGlres/+LuSKcKEgTn5ljg",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-25T12:46:33.824245Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-25_05_46_32-11226913448634491822'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0825100258'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-25T12:46:33.824245Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-25_05_46_32-11226913448634491822]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-25_05_46_32-11226913448634491822
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-25_05_46_32-11226913448634491822?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-25_05_46_32-11226913448634491822 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:36.433Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.060Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.097Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.261Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.300Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.406Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.455Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.490Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.521Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.555Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.589Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.624Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.660Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.706Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.741Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.775Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.822Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.859Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.898Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.930Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.964Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:37.992Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.072Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.113Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.155Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.190Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.226Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.386Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.452Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.501Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.543Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.635Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.660Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.702Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.731Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.773Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.808Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:46:38.877Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:47:07.697Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:47:07.736Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:47:13.042Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:47:13.101Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:48:47.396Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:48:47.436Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.420Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.492Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.525Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.595Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.630Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.653Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.667Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.686Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.723Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.740Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.756Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:52:02.817Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:54:17.955Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:54:18.173Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:54:18.232Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:54:18.312Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:06.009Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:06.088Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:06.143Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:06.220Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:08.934Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:09.001Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:09.172Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:09.233Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:09.237Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:09.286Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:09.304Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:09.353Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:09.422Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:12.271Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:12.343Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:12.414Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:12.469Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:12.527Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:12.592Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:15.365Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:15.434Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:15.507Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:15.682Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:57:15.719Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:58:04.818Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:58:04.853Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T12:58:04.888Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-25_05_46_32-11226913448634491822 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598359590048/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.061821937561035156 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: e9430d50f7874c1ba34fd3e5f14d3732 and timestamp: 1598360390.601246:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 800
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598359590048/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598359590048/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.059340476989746094 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 801.156s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 23.823 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 33s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/mcc5b6jnee7sc
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1843
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1843/display/redirect?page=changes>
Changes:
[noreply] [BEAM-10777] Add two blog posts detailing changes to the type hints
------------------------------------------
[...truncated 291.01 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWZpMUXMACKN/////////////////////////////////////////////4AjT6gNslgjZtCxsgE9YVE4g0UIZKYniNI9TTEeo9JtJ6nqNPSZGQ0A00DT1AAAAAAAAGgAaAAAAABoGhoABoAGgxGRoNNDQyDR6QZIINEZMEIyaJNqNDRppkA0GgA0Bo00AAAADQGgAAAAAAAaNAAAAAAAAAAAAACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQep/qkGITACNMTTABMEaZMBGmmTAEwBGACYAIwARppiYATAAAjTAAJgAAmmAAIMIaNAwAJMABMAATAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASJE0CE0yTIyNU/I1T9CJ6jQeKB6jyj1NqaZNMRoADQDTQNGQZPJqaAaAAADQAAANAAAANAAAAAADQZVMO1upjAml38WGXa9MjhRO4Tx2bNrEx4JktF5kUKxdXJrD1eu6/v33msphXMrQzfUUugqqiXQ6IQmTAumsEQgIUKBASNMSbSAZpZ54CaS5RMY13fwz0yU2MaEpiEE65kkAYs5YLAUCeJCPgWhAF8KK9yxhUXb163gRX1/czbSwlppGExGYdF/k0UjxRsAicC02aJszZGmcZm6QaBbeXHRkWi3kZxzpMwHRniSQWGIMXgSs8XeGK4mVorxkg8FTb5SsK50JhMJwTkKVpKYTAnYT8jNTlARz3g44nQdC25znJSr4sVpubOLI0LIZh56olOkIvv4d068hiCWKAjGShyUSJSGwmaJKSGxvIQQ2HGkKNsSGSz3NMhtMjGcmM5QniM42IVGgTLVmhnmkULCZYUt62FoQzmoxM5VVHYgkZUYOl2HcsH/q1eoquzDa7PEW/ucBxe1x+2kBJsC3/U935oRb2OyUqwjoAQHO8UbexJnXLjagjcDvfN870HRdN1WYeeeeefa1jGMY1rWMWv/ax0CoZ+EyAYN4y28t0fI6dkPQ8DndVcjI5Iu7GKoXTxNNXsLlkJzZMTdTIWbxMypWCt4ShUVahYK7VMOf86uGufsVessETjqKlw5cTQXElJu/VcOqFWhC6HwF8CLyp5lUJIaTQViyUjCasUUEQbYoXcpuW5hQyYa+MqIHV40TDw+OGILpobSLjIicoLYojBOUE7WFSiApXaoFMzVy5xIN/0C3Raps20zyBImcmGG9rvwSR6h6LzbGWdhhkViWKd3hRRvfOzLzrZMtOv67Hzm41CpcDS2ZM2YZokqVAzjY3P6rWbBPa1Dc4/lZCS7kpL+yZ+89CvgMzBisMmVVfWr8g/k/kyRJLTJKKGrNShRiLkLQask7SUSCQLW1upxtg1ki5VRRr9uK/zOH0NVHUjfModbGeGXeeLEXN5j7kxutUuKvWudHrza3JOCTQkDTaoJO6G/REfRkMjpnyjN73dy2WoU70q4aVTxy7EJEGDHCI1IuMSD8L+ScSZxBECh4iMNYSyrPoJQJefh1stzJqKm4i+UCk1v5Vh1CkyosiqV/AjBmitUqy+JO4nPcTzdMpvitdcVbCyY8qm0Cyc1eVlWCoGF/M282jl0xWpJWKsMQshmrALsAK5ibN0pZzqA++DMHIJchndMhephiWkQZG2cyvmobV4FsKzKXzyEKjMRAEoTEkiTA78p9t0piQBufBOKBaZB2QIPZVrpdaxdWF3jPGIx/LkEaVzV3YMdqiAVPn5PodKRVk4I2jJwVMpqogCcAGqEUuCYBxABRGdIvImqEpF/JFZonayQw6QoHojwtpr66qTUsctUx1zQgNhdhbECY1AYEBAppeJoBxRrEGUQKdIZInEsbdu7xbx1HNl5Bj39qsFjXHLfK5VN30XWVKtRVgWJAqUth2VihVzdUm66zQSSwr/tt1pMxkcAndvqyueKXXQiwMyRqlNGfMCpeLZ0yso6iV0RMi/YNA6TQRyqgmBsWCg0KE7VjcYBU49VXF49mFdKPotY2gF25TgGikLXpKgb+xbQrFwF9QEoODOVsE4DQsLfrpS+TBhgGr2KQnXCz9Fk5haIYGIEvR7ZKbhyDsIYQQUAvAYNYRfcLmqJmCkpX3TUT0Zt3IAmjShr6clAw1shKk2WWcmIKagW654HzMhZAJWc98+5DYByAYFmAnpmbdWcB/UC+cMPT2+Rdffae2JsRRYaGA/AcK/jzg2gpHCavZyAyy7eq3sJyw0Ns08ssgRmmS0VrN9PnxYMuq1DvVrKERIbU4jjr6lSQdoLGlJxaU6yIw4C5jAbGTGNZgsWCoXjKcdCQbVwoYUXhM1sroxEpXknmYRwEwFCWoGhUGoZ1+jbTGECsYoveEeK7gJ7DMsK7IasEmwSytTUlnK4GTUwZRNu8pnwTFjxEWnIlio1xVVdwcAsGrtKgsGjMaKWBgCpXlrU0J0IIqgDE5Cn7fDMTbd7LoKMJKjVwohzLkxkmoIV29Mc555qQbnls8UsXdtZi2SQRsTXNDQKGWxWTTtDCSUNO0qhT3GXJLPdvGpmYVZVRIur2sZMatNvbUa4ijV6MrNqtJAsZ0KlhhsGzVgKLBtIsSsjDGxUrM1gNmwLU201Wu8ev69SWdjIalzkPkOciIqgg1Fa15ou3mIGOgggrMgha2W2iQzpJDDDCYa77F0qauDfUYiPSyyKWlvy2Mr+5NBcc2w81j8S75qmudbTEpE5QkjUGVWntQbGVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEERKzlLSmxCKaXOfUqBaIiIjnKWjnODnRRRPTrEccy3o4i9yyZMyTbbbdVWCVcqpNtt3ZspcNXTnERgySlSaLWLLhkbLMKtBpSmlttbUnhTWkJjGaasNU1ICr7Wng4x27vFiBSErS/brcZcydt1Y7IS45gosHGsrRUwYyDmAzzQNJVi3BR7kc2NS3a6rhJLM0dfdn52+zwfhC8Z3xWBgGPqB0Q4vHB9BjzJOZ2zCcVKsrkFlL9+ctBgGWBYcOWQZHr63hyo2Yz04ztAbnNMZglvGlMJyy6+KrI+JoAUyLzgQs0IoEQGyrPqpbjBVBcD4Kq5cXxALE0z6BMUY60onjnfUEGViutfe9lz+UCIn4MUsVOw71BNFZ6OkrAQQSnCtCmIRIRbwfb7DMzfL9f86nURipgTQvGT4+ktqAvYhtyNVaXB0UjRV9CRiRst0pI0q9MlKxbW00yhmyW2He4YNq4qtYSrKsRt1VSllxWsG7U7TetDZNdKxTWAjzlPDsKUMMJEGEhBgLIMGfplpaQefVWW0BKRcp7PbWJn3/D416a+S6bXfP6fWcV/ZWx/E7Pfn9fyf/v+dUDe6iCE7VUBGAOQAQbhGEEGFADAMJBIKwDMCzX6iif/tpRAdIRP+inSNA7h06oLQtBngA9D1JvQSiD78ftGSBsGmDYNMGwYvuxAbEHQggUhoBtJDbaB9JAjmWg8u0KwaWp6z/4u5IpwoSE0mKLm",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-25T06:46:28.183549Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-24_23_46_26-17649069668830347684'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0825050320'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-25T06:46:28.183549Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-24_23_46_26-17649069668830347684]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-24_23_46_26-17649069668830347684
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-24_23_46_26-17649069668830347684?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-24_23_46_26-17649069668830347684 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:31.527Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.156Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.238Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.312Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.344Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.440Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.483Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.517Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.544Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.590Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.625Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.650Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.678Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.713Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.750Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.784Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.820Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.856Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.893Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.928Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.961Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:32.997Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.035Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.073Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.100Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.143Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.178Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.332Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.389Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.413Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.434Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.447Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.471Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.529Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.529Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.587Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.623Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:33.659Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:46:43.742Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:47:06.599Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:48:39.962Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:48:39.995Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.515Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.600Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.629Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.720Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.753Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.776Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.781Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.816Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.835Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.852Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.888Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:51:50.916Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:54:08.288Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:54:08.385Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:54:08.441Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:54:08.515Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:47.586Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:47.667Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:47.728Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:47.797Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:49.909Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:49.994Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:50.069Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:50.105Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:50.120Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:50.161Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:50.187Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:50.227Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:50.292Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:52.844Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:52.931Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:53.012Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:53.063Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:53.131Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:53.209Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:55.562Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:55.639Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:55.852Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:55.918Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:56:55.953Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:57:49.641Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:57:49.684Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T06:57:49.719Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-24_23_46_26-17649069668830347684 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598337984285/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06971955299377441 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 4bb177c5241b4655ad12398c7363004b and timestamp: 1598338759.799575:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 775
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598337984285/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598337984285/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05524754524230957 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 775.997s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 12 mins 58.659 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 1s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/mqxd3thj5tquk
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1842
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1842/display/redirect?page=changes>
Changes:
[Robin Qiu] Propagate BigQuery streaming insert throttled time to Dataflow worker in
[noreply] Merge pull request #12415 from [BEAM-10603] Add the RecordingManager and
------------------------------------------
[...truncated 292.31 KB...]
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWeKoJmUACKN/////////////////////////////////////////////4AjT6gN24uCNm0LGyCk9g1E4g0UIZKYniNI9TTEeo9JtJ6nqNMjJkNANNBk9QAaAAAAAANAAaAGgAAANA0A0AGgAaB6hoaDTQ0Gg02kGSEaBMTBTIyaE0j1DR6mm1AAAAyaNGmQAAAAAaZPUAAAAAAAAAAAAAAAAAAAAAGSNigAAAATAJgARgBMjAACYAJgAEGBMRgEwmTAAAAAAAAATAmjIwIMAAAJoQep/qkGITACNMTTABMEaZMBGmmTAEwBGACYAIwARppiYATAAAjTAAJgAAmmAAIMIaNAwAJMABMAATAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASJE0CEwgjI1T8plP0JHqNB4oHqZqPUyaYjEAANANNA0ZGmmynqaAaAAA0NAAAAAAAAAAAAANAAGXTG2N1MQE0uaFdluOmSAUTwCeK6urGDFfGSwXWQvrFtcmUdjjgvLxrrjKX1zK0Q3dFMD1VUK6IIQ+ZL22awRCAhQoEBI0xJtIBmponf5pLlUxjXdugeqSmxjQlMQgnXMigDFneQsBQJ4cI0CwHgvgxHar39Fu5cs3sN3eWs31MJaahhMRnHRf4NNI8SaAROBarNM2hszVOLyagaRbeXHSEWi3sZ5zpMwHSHiCQWGIMXgCs8VeGK4mVorxkg8DTcZisK50JhMJwTkKVpKYTAnYT8hNTlARz/go4fhehbc5zkpV8SK03NnEkaVkMw8/USnSEX38K6deUxBLFARjJQ5KJEpDYTNMlJDY3lIIbDpxCjbEhktFzTIbTIxnYRnKk8RnmyCo0iZasgaJqFCwmWFLetBaEM5qMTOWVR2AJGZGD6mh3HB/6tXiLrvrbXR4jH9xfpPr8TsJAS7At50fd+qEW6kcg5YSMAIDmd/K3kS51y32kIx4j2S0xNztDS1IYYYYYsEECCCCCEEECCJE944IRCUBDAFEcAEB79PqCSkxRR+j4HO665KRyZdygqhdNE01ewuWUnNkxN1MhZOHnVF4F/USoulpUYmvlgFzmTmjmHi3GHFrLczK8MsVi2DEsXiWjkirQg1y2IS12tKpmKqdIzQZysVFIwmTKKCINrULWV3HcgoZQONGWD0EcSJg4PGC/MEz9hFxEQwKC2KIuHKidxhUogKV2qBTMi7IcKDjeGt2WqbNeZ48kTOUDDe13wJI9E9B49d6HYYNFYlh3d4UUb3nrTH5pqZaVf1GPkOF0ypcDS2BM2YZokqVAzkJDOaiS1qeyqG2xkpjZbs+eY9kz952pvgMzBisMmVVfWr8g/k/kyRJLRpKKGrNShRqNkC0GrJO0FFAkCkq3DyFg1PLsiqLNftBX+TwJXT6CkbtlELY7vTDzJxNweeFb2Nyql2VyTiR1ZtbgnBJocc02qCTuhvERwqMhjtG+UZvd7qWu0qnbcwjSqeCYYhIgwY4JGpFwyQfhfSTijOJIgURExxrCWVZ9BKBLzsOtludNRU3EXygUmuOlWHT6TKiyKpX8CMGQVqlWXxJ3E57meTVKb8rXW1WwsmPMptAsvNXlZVgqBhfzNvNo7KmK1JKxVhiFlMiwC68CuYmzdqWe6gNaDMIEEujZ5TIXaX4VpCGROTmVpqG1eBZCuuS7dQhURhIAlCYkkSYHc8/22ymHAG17s4oFpkHZAg9lWuf07F0qnuGeKRk+Rx0aVyl3VMdeiAQ/NyMrz+UrKEI2jKEKmU1UQBQgBqhFrgmAcSAURnyLyJqhKRfyZWaZ2okMPclA6M8HabGwqk1LHLXMUZoQGuuutiBMegLyAgU0uk0A4RrEGUPKgkMkMBLG3bvMW9dRzZeQY+OtVgsa4xb9XKpu+i6wpVqKsCxIFSlsOysUKubqk3XWaSSWFf9tutJmMjgE7t/WVzxS40IsDOka5TTnzAqXi2lMzKOoldETIv0GgdJoI5ZQTA2TBQaVCdqxuMAqceqri8e0CulH0W3NqBbtU3popC16YoG8r2UKxaBaoCUG+nK2CcBwLC2xpS+LBhgGr2KQnXCztFl/WtEMDECXl9stNxBB2EMIIKAXYMGsKvj7iqJmClnPumoqVm3cgCaNKGvpyUDDWuFqTZZZiYgpqBbrngfMyEaA5nPfPOA2AcAGBZQJ2Zm3VmAf1AvnDD09vkZX32ntibEUYGhgPwHIX8ecG0FI5Bq9nIjLLt8revnLjQ2zVzSyhGQy2isZxqU+OBlydQ7xayhEKGxOIoo9VpIO0FjhScWFQXQiEAFrDBr6yMey+YsFQuGU5B8g2LRQvou/ZliujAlK8i6y/N6mAoS1g0qg1jPv07aYwgVjFF7wjxXcBPQzrCuyGrBJsEszU1JZ6uBk1MGUTbvKaMExY8RFpyRYqNcRVXcHALBr7WoLBozmilgYAqV5a1NCdCCKoAxOQp+0wzE23ey6CjCSo1cKIcy5MZJqCFdvjHOeia0G65fRFLF3XbsWzSCNkbBpaRQzWF5hR8CsFlSFHw4oswp5l1I/nRQSBycukiBuAqaUvm4tqNgRRq9GZnFLUQLGdCpYYbBs14CiwbWLErIwxslKzIwGzQLU201Wu7+t65SWfjIalzkPkeciIqee1Ra14xdnrHmQPPPVmPPuOS2UR+dJH333y/G1i6VOLezCMRHZZZFLSz5TGR+5M9agcruuMbCu7cU5BBZTBSGBQkiUGVWHdTciKh4eJhAGDCUrFAAAAAAAAAAALAAGgAEYsWTChoEACCCCCCN4bok9DJaaGhddXEWSbbbuuiTuuuLrr776ZpGDBllTBEXuWXLnSbbbbqqwSrlVJttu7JKXCV05xEYMspUmi1izYZGzzirSaUppbbb608Ka1BMYzVVhrmtAVfa1cHFu3eYsQKQlaX7hbnNnTturHZCXGMFFg426tFTBjIOYDRNI1FWLclHuhzY1LeLqmEks7T2N4fnb7NB+ELhnaKwLwxKgdFOOyAagxDJOZ3JhOKlXS5BdJeNnLgLwzQLDhzSDK9jb+GKjaDPSjOzhushjMEt60phOWbYxVZWiZ4FNLdgBCzQigQ8bpWf1S2mCp60GgqjMBaHhXmmagTCMVaUTxTtUEGXhtuNd9iButhET8GKWKoMbcoJorOxUlYCB6U39gUwiFCLN97XYdfN8v1/zkqiMKYE0Lw58SkuVAXMBy5GutTg6aRpq+hIxI2e7UkaleqSlYuKtNUoZJLbDvcMG1cVWsJVlWI3CqpSy4rWDeKdpvmhsmvpsU1gI81Tw7ClDCDhgwkIMBZBgz8rlzKPPprLaA80Z5vZ7axM+/4fGvTXyXTa75/T6ziv7K2P4/M7+v3/Z7/T7qgb3MQQ/FVQEcA4ABBuEYUQYUAMAwkEgrAMwLLfKKJ7/WhEB0hFT8W6RoHcT8+6s60Gd+D0KBN6CUQffj9oyQNg0wbBpg2DF92IDZA6EECkNANpIbbQP3UCOZaDyzQrBpa3qv/i7kinChIcVQTMo",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-25T00:46:34.523788Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-24_17_46_33-8733151788777511898'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0825000824'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-25T00:46:34.523788Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-24_17_46_33-8733151788777511898]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-24_17_46_33-8733151788777511898
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-24_17_46_33-8733151788777511898?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-24_17_46_33-8733151788777511898 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:37.404Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:38.864Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:38.902Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:38.974Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.015Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.109Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.151Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.173Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.199Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.233Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.260Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.295Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.330Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.364Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.401Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.423Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.457Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.481Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.517Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.541Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.572Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.608Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.641Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.670Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.722Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.751Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:39.782Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40.065Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40.101Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40.112Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40.134Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40.144Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40.202Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40.202Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40.256Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40.289Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:40.326Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:46:47.616Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:47:14.934Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 6 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:47:14.964Z: JOB_MESSAGE_DETAILED: Resized **** pool to 6, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:47:20.410Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 7 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:47:20.437Z: JOB_MESSAGE_DETAILED: Resized **** pool to 7, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:47:25.992Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:47:26.022Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:48:15.028Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:48:51.335Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:48:51.371Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.696Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.757Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.778Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.847Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.870Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.896Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.897Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.912Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.945Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.954Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:09.986Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:52:10.011Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:54:53.216Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:54:53.293Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:54:53.351Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:54:53.412Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:42.533Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:42.594Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:42.642Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:42.698Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:44.918Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:44.978Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:45.082Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:45.106Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:45.126Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:45.154Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:45.192Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:45.228Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:45.310Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:47.679Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:47.751Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:47.809Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:47.880Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:47.929Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:47.986Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:50.703Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:50.768Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:50.852Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:50.953Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:57:50.986Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:58:42.066Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:58:42.106Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-25T00:58:42.133Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-24_17_46_33-8733151788777511898 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598316390786/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.07180428504943848 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 9a25ee0023924912b21d30d4fabe8c77 and timestamp: 1598317224.2084987:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 833
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598316390786/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598316390786/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.058439016342163086 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 833.914s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 56.385 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 15m 6s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/wjpi3hlzuqpqe
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1841
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1841/display/redirect?page=changes>
Changes:
[Udi Meiri] Silence red PYTHONPATH warning from tox
[tobiasz.kedzierski] [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac
[Kamil Wasilewski] [BEAM-10524] JSON decoder for ReadFromBigQuery supports repeatable
------------------------------------------
[...truncated 289.67 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWXPJSEcACKN/////////////////////////////////////////////4AjT6gNtFgjZtCxsgpPWFROINFCGSmJ4jSPU0xHqPSbSepk0yNGQ0A00DT0gBo0AaAAAAGgAaAAAAABoGhoDQGgAaDEaGg00NDINHpBkgg0Q0wQjIEaj1DTJpkA0GgaDQGjTQAAAANBo9TQAAADQAAGjQAAAAAAAAAAAAAkwAEwABMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRNAhMJMjI1T8ptU/Qkeo0Higeo8o9TJpk0xGgANANNA0ZBpmmpoBoAAANAAAA0AAAA0AAAAAAAZZMGzvBjAlkwxXZcq0VnCeZwmjsWLOJjvjWsl5kUKxcXW1t6rVdd3b7zWUQrlVo5up6HQU0xLndEITWvLhqBEICFCgQEjTEm0gGaegd/mkuTTGNd2556hKbGNCUxCCdfyaAMWcuFgKBPFBHwLIgC+GFW3Xv57l7e2ryK6u7ed6eEtNMwmIzjov8GkkeJNkETgWozSNqbQ1Di8mmGiW3lx0ZFot7Gec4TMB0Z4gkFhiDF4ArPFXhiuJlaK8ZIPA03GYrCudCYTCcE5ClaSmEwJ2E/ITU5QEc74KOH0HQtuc5yUq+JFabmziSNGyGYedqJTpCL7+funXlMQSxQEYyUOSiRKQ2EzSJSQ2N5SCGw6cQo25IZLQc0yG0yMZyQzkyeIzzZhUaJMtWQNA0yhYTLClvWgtCGczGJnKKo7CEjMjB0uy7jg/9Xr3EWH1tru6RL23wHF7PH7SQEswLfynu/NGLeyGSmGEg+B853hj74JY7Bc7aEYw79mud6Doum6rNvPPPPPtaxjGMa1rGLX/tQ6JSNDCZAMG95ffG6XkclkPQcHnNZchI5Eu7EKoXTRNNXsLllJzZMTdTIWTh51RgBhyRUXi+VGN+EsAr5o4ZzEBfxBjVhtXiyzD4JlkpdZlTFqtnVyrRheDMlzIa6ppFTpWNBnKxYKVAmslFBEG3qFzK7ruwUMoGvjLCB1WNEwcHjxgF0sNpFxkROUFsURfHKiZrCpRAUrtUCmZFyxw4OB0C3Zaps15njyRM5IMN7XfQkjKHovLsZlyGGRWJYpzdFFICO7E1P1FMtOv6zHzHB6hUuBpbMmbMM0SVKgZx8fn9VrNgntanucfyshK9uXmvZN/egqzwDMwYrjJlWXlq9IPZPZMkSS0ySihqzUo0Ui3C0GrJO2FDgkC1tfq8fZNZJxaqINftxY+Zw+hqpCmb5lCrYrwTTrxYaNdZGNY3asRivWt9Hrza3RN+TQkjTaoJOaG/REjSkMjpnilP8deC2WoU7sxBNKx4ppiEiDBjhEaqXGJB6F/JOHM4YiBQsNFGsJZWnkEoEvPQ62W501FTcRfKBSa4Eqw6fSZUWRVK/gxgyCtUqy+JO4nPczyahTfla62q2Fkx5lNoFl5m8rKsFQML+Yt5pHLJitSSsVYYhZTIsAuvgrmJs3alnuoD74MocgkyGLorF6iGJaRBkbZjI+aRtvgWgrOJdPIQqMxEAShMSSJMDvS/23SmJAG5788oFpkHJAg7lXul1bN1KtvGeMRkeXJI07mruuY7NIAq/PyfQ6UmrKCI2jKCKoVFUQBQQBqhELgmAcMAURnyLyJqhKRfyJWaR2okMOkKB6E8JabGwqk1LHLWMdUzoDXXXWxAmNQGBAQKKHiZwcI1CDIIFOrGtE4ljbt3mLeuo5ovIMfAtVgsa4xb9XKpu+i6wpVqKsCxIFSlsOysUKubqk3XWaKSWFf9tutJmMjgk7t/WVzxS64EWBnSNYppT5cVLxbWmZlHUSuiJkX7JoHSaCOUUEwNmwUGjQnasbjAKnGqq4vHtQrpR9Frm2AuW6Lwz0Ba9NUDd17SFYtgvqAkBvpitgmAaFhb9VKfyYEMA1dxTE7AWfpMt61ohAYfS8/tlZ2FIOQhA/BQC8hg1g18ZGqiZgpWY905D9GdcyAJo0o6+pJPsJb4OqtllqJiCnH1/XOg+ZkLQBMT3voW4bANwDAiwKCbnXFqAe1AvnjDs7vETY3mAbE2IokNDAegOOv404VoKRxzV7OPGWXb5W9gOVGhtmpmllCMhltFZzzUpsUDJrFI71ayhESGzMI46uq0EHayxpSYWVOsCMOAt4wGvlBjWXzFgqF6ZDjoSDZtlDCi79mvFc+BIV5N5l+bxMBQlqho1BqmffpW0xhArGKL3hHiu4KeyzrCuyGrBJsEszU1JZ6uBk1MGUTbvKaEExY8RFpyBYqNcRVXcLALBrbaoLBozmilgYAqV5a1NCdCCKoAxOQp+2wzE23ey6CjCSo1cKIcy5MZJqCFdvjHOegasG65XQFLF3XXYtokEbM2DR0ShmsMDCkIFQLKkKQhqizCmlXSkOfFBWDZi8SHP4CpZDCbi2o2BFGr0ZmcUtNAsZ0KlhhsGzWgKLBtosSsjDGzUrMjAbNkWptpqtd39d12ks/GQ1Lm4fH83ERVBBqi1rxC7WbgY6CCCoyCFrZLSJDMlaGGGEw1X2LoU1cGYRiI9JJWUtLXlsZV9yWC25td5rH4l3TVNc60mJSJyhWjUGU2XtTbGVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEERKjlLSivCKKHOfUqBaIiIjnKWjnODnRRRPTLEccq3o4i9yy5c6TbbbdVWCVcqpNtt3ZJS59XTnERgyylSaLWLNhkbTOKtFpSmlt9fVnhTWmJjGaisNY1YCr7Opg4t27zFiBSErS/cLc5s6dt1Y7IS4xgosHGurRUwYyDlw0DRNNVi3JR7oc2NS3i6phJLO0tjeH5m+zQfgC9Mz4qAwDH0g6QcVjg+gx5rTGZsomFKrC6wsJdvzFoMAzQLDhzSDK9jX8MVG1GemGdnDdZDGYJb1pTCcs2xiqyviWAFMi84ELM6KBEBsK0KmS2wUwWw+CqqXF8QCvLK+gTCMdSQTRzPqCDLRXGvvey5/KhET8GKWKXYN7OTPUejoKwEEEhv7IoiESEWr72+wzkvy/X/OrUkYUoJnXjJsfQW0gXuA25GstPhaSRpK+hIxI2m7UkadeoSlYuKtNQoZJLbjvcMG1cVWsJVlWI3CqpSy4rWDeKdpvmhsmulYprAR5yADkKcMMJMGEhAgLIEGfomZmSdfTXWz5LxMv7PbXJv3/D42Kc+S6dXfP6fWeV/ZWx/E/Qfn9fyh/v+cUT/dg/CfrKAigG4AIP4RgxAhQAwDCQSCsAzAi79SRQ/22IgOEIf/RDhGicw4cUVrWgzwAdh6U3YJRB96P2DJA2DTBsGmDYMX3IgNmDoQQKQ0A2khttA+kgRzDQeWaFYNLV9X/8XckU4UJBzyUhHA==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-24T18:46:29.209972Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-24_11_46_27-14398290832772796054'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0824154410'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-24T18:46:29.209972Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-24_11_46_27-14398290832772796054]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-24_11_46_27-14398290832772796054
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-24_11_46_27-14398290832772796054?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-24_11_46_27-14398290832772796054 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:31.607Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-b.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:32.845Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:32.889Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.079Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.107Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.213Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.264Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.290Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.325Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.358Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.393Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.420Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.457Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.491Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.525Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.561Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.596Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.633Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.665Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.692Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.725Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.760Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.791Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.825Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.863Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.903Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:33.945Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.158Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.222Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.250Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.261Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.287Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.297Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-b...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.354Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.354Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.412Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.448Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:34.521Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:44.468Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:46:58.421Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:48:34.443Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:48:34.478Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:12.792Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:12.864Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:12.906Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:12.971Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:13.005Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:13.031Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:13.043Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:13.053Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:13.083Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:13.094Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:13.105Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:53:13.160Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:54:11.267Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:54:11.334Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:54:11.387Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:54:11.436Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:49.991Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:50.046Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:50.090Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:50.159Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:52.675Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:52.724Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:52.779Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:52.817Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:52.835Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:52.870Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:52.895Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:52.931Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:52.993Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:55.596Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:55.668Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:55.730Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:55.777Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:55.840Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:55.906Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:58.517Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:58.581Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:58.695Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:58.830Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:56:58.859Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:57:50.327Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:57:50.419Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T18:57:50.451Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-24_11_46_27-14398290832772796054 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598294785142/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06347227096557617 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: c11658f2839248a2a722f22304ed7cd0 and timestamp: 1598295583.104606:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 797
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598294785142/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598294785142/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05551004409790039 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 798.554s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Daemon **** Thread 3,5,main]) completed. Took 13 mins 21.7 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 23s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/tmabp4jjocx2e
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1840
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1840/display/redirect>
Changes:
------------------------------------------
[...truncated 290.79 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWcH7YY8ACKN/////////////////////////////////////////////4AjT6gNtFYY2bQsbIBPYNJOINFCGSmR6aNI9TTED0m0nqaNMTRkNANNA09EADQAGgAAA0ABoAaAAAA00A0NADIAGjTahoxDTQ0Gg02kGSCARkYgmJo1Tyj0jR6mm1Gg9QAaD1GjRpkAAaAAZANNAAAANBpoAGgAAAAAAGgAAAAAAkwAEwABMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRNAhMIIyNU/IU/Qieo0Higepmo9TJpiMQAA0A00DRkGmynqaAAAANDQAAAAAAAAAAAAAAAZRL+tvRiApyZgUmWqM8cAmlgEsVWrWvorwx1i4yF9YtLjyLlGjBYsOuNZO+umrSzdTTwPVKkK5oIQ+Y7u0aARCAhQoEBI0xJtIBmponfJpLl0xjXc+heqSmxjQlMQgnYsRAGLOXCwFAnjAjoFYPBfECjZpXs1q3br3cN1Ys6BqYS01DCYjOPGf4NNI8OaAROBarNM2hszVOMyagaRbeXHSEWi30Z5z5MwHSHhiQWGIMXfSs8ReGK4mVorxkg79Tc5isK50JhMJwTkKVpKYTAnYT8fNTlAR0Hf44ng+ibc5zkpV8WK03NnFkaVkMw9BUSnSEX38O6deUxBLFARjJQ5KJEpDYTNMlJDY3lIIbDjSFG3JDJaLmmQ2mRjOvDOXJ4jPNkFRpEy1ZA0TUKFhMsKW9XC0IZzkYmcwqjroSMyMH09Dt2D/1iv8NZfa0vdJh3lrGSEpye4kBLLy4XWnP1SC4Edlmy+Oeg9dHx5vACWOyXe4BG7Hf81z+bznTdVnXHHHHHWtYxjGNa1jFr/2gcdUGkBMMMGWy++t0/Ucfh9Hwef11ykjlS7lhVC6WJpq9hcspObJibqZCycTOqLwL+PKi4XSoxOvlgGYzR4Q8E8Xb4xKZwa8EsfW9UKhXS6OIVY4O2BCbLzjczDimZlJkjM5mKxVKRhMgUUEQbgoWslu27hQyIa6MmHoKMSJf3/Ii4YKb9hFxEQwKC2KIvDkhK1hUogKV2qBTMi7CcODj+It4WqbOlTPHEiZywYb2u8hi/FTk89m/eiG8rdZlqFF9/ByvlfD+mhPHmaXkT+ymj802QIzOG5cFxZaqBnGxui10fs09xWN5kuZk5XuzEz7pv70FadgZmDFgYsa67s3hB5J5JiiSWpSUUNWalIiEWoWg1ZJ28oUEgUfY63G2jX9CKVQxuW6Fn5/Gk9bpqfwWMGtiPFMuchCRbnyLuwvVci1ce1jtibO8JviaHNNNmgk4ocJEcimoZTUu1NfcDfC2mrU77Z/ZVzyTLAJEF7DEIwJSJIPAwhJwpnCEQKDhIg1hLK87glAl5yHWy3OmoqbiL5QKTXISrDplJlRZFUr+DGDIK1SrL4k7ic91PJqlOAVrrCrYWTHmU2gWXnLysqwVAwv5u3nUdiTFaklYqwxCymRYBdbBXMTZvFLPdQHXQaYgQSYeiTxlyd+FaQhkTZTI6agyvg1wrr0unEIVEYSAJQmBJEmB4Jj7bxTGgDd+GfUC0yDggQdSsXU7Fq69a31RyCMhzuajUpJd2jEpSwK30svJ9SSVk/kbJi/lVKqqIAn8A1QhlwTAOEAKIz5F5E1QlIv5UrNM7OSGHuCgdGeBtNjYVSaljlrmKiZkBpLpLYgTGIC8gIE87hMwOAaBBkDyoIzHDASxt273FvnUc6XkGPkLVYLGuOXAVyqbvouqqVairAsSBUpbDsrFCrm6pN11mkklhX/bbrSZjI4JO7gVlc8UuPCLAzpGuU0580Kl4tpTMyjqJXREyL9BoHSaCOYUEwNkwUGlQnasbjAKnIqq4vHtArpR+N3BtQLVme7M04WvUlA2KVdCsWQXVASA3kpWwSgNCwt2ilT5T8F4Zuop6dkLSUyW9q0QQMPZS81Kz0GQcBBB8CgF9C9nAOV2i1RVEFKtpydhZOecSAJkypTlVknuCuUBAtFkUTAFOvb6upI+ZkLYA2n/fQtQ0AagGBEgUE3PN4oB5UDlPmHR1doezu9XaE0IocMjAeAOSv5E4VoKRyTV7OTGWXb9W9cOZGhtmrmllCMhltFWxWry4sGTIVB3q1lCIUNaURRUdanIO3FjSkorKgqiIQAWcEGlr4xjLxiwVC2ZDjXyDWslC+i71mTK5rkhXiOMvTdpgKEtYNKoNYz79O2mMIFYxRe8I8V3BT0M6wrshqwSbBLM1NSWergZNTBlE27ymjBMWPERacoWKjXFVV3CwCwa+1qCwaM5opYGAKleWtTQnQgiqAMTkKfs8MxNt3sugowkqNXCiHMuTGSaghXb8xznomtBu+Z0RSxdy3DFs0gjZGwaWkUM1ismnaGEkoadpVCnus2WWi7eNTM4qzKiRdXtoy41abm2o2BFGr0ZmbZaiBYzolLDDYNmvAUWDaxYlZGGNkpWZGA2aBam2mq13T1nWaSz8ZDUueh8nz0RFTz2srWvPl1868yB556gx59rZK6I/Kkb7775fousXOpq3uBRiI5JJGpaV/LYyj7lN6zA2k41jsK7pqmwQV0vkhgUI4lBlSs5rDYioeHiYQBgwlKxQAAAAAAAAAACwABoABGLFkwoaBAAgggggjeG6JPQyWmhoXXVxFkm227rok7rri66+++maRgwZZUwRF7lly50m2226qsEq5VSbbbuySlw1dOcRGDLKVJotYs2GRs84q0mlKaW33GtPCmtQTGM1VYa5rQFX2dXBxjt3uLECkJWl+5W6zZ07bqx2QlxzBRYONwrRUwYyDmg0TSNRVi3RR7sc2NS3q6jhJLO09jen5m+xwfgC2ZXRQBeGFUB0w4vGh1BhGOUytpiUVFVVxiqlh2UtBeGaBYcOaQZXsbjwhUbQZ6QZ2QN3kMZglvmlMJyzbGKrLWZbBPg0uESWe4EWLSje6OPaTNGzaFYoxK5Vlhky5XUCYBioSCWKV1QQZSG011zPQO7CERPwYpYqQX9uYmag5FOVgIHpDe1hPCIUIr3ntdj2FP5fr/nW6hGBTBMy8GXCnLagFu425GutThaaRpq+hIxI2e8UkaleqSlYttaapQySW3He4YNq4qtYSrKsRuVVSllxWsG9U7TftDZNfSYprAR6CrhwFSGKEiDCQfgFj8DOXbNuY5+uwtHr0w/p901YJv3/D42ad+S6eXfP6fWfV/ZWw/H5oP1+/7Q/6ib0b7eg+D8V1ARADUAEH0IwAfgoAYBhIJBWAZgRODpgof9b0QG6ELRQzdGjcRRUTe3LQZxgdBRpugSiD70fsGSBsGmDYNMGwYvuRAbIHQggUhoBtJDbaB+5gRzbQeVaFYNLW9T/8XckU4UJDB+2GPA=",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-24T12:46:22.666171Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-24_05_46_21-4734129608145900368'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0824100317'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-24T12:46:22.666171Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-24_05_46_21-4734129608145900368]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-24_05_46_21-4734129608145900368
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-24_05_46_21-4734129608145900368?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-24_05_46_21-4734129608145900368 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:25.226Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.065Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.105Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.175Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.196Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.310Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.347Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.371Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.400Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.422Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.450Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.472Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.499Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.532Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.568Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.604Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.639Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.671Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.705Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.729Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.753Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.788Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.824Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.860Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.897Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.928Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:26.962Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.291Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.358Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.382Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.393Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.409Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.419Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.481Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.481Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.548Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.581Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:27.617Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:46:40.441Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:47:00.095Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:47:00.123Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:47:37.084Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:48:28.222Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:48:28.259Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:05.776Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:05.873Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:05.907Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:05.975Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:06.018Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:06.042Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:06.050Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:06.056Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:06.112Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:06.119Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:06.145Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:06.274Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:09.750Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:09.817Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:09.869Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:54:09.937Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:48.484Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:48.546Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:48.595Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:48.664Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:50.821Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:50.895Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:50.958Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:50.985Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:51.012Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:51.034Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:51.088Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:51.122Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:51.192Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:53.908Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:53.985Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:54.053Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:54.107Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:54.176Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:54.245Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:57.206Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:57.285Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:57.419Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:58.126Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:56:58.160Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:57:49.184Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:57:49.225Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T12:57:49.254Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-24_05_46_21-4734129608145900368 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598273178688/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05425429344177246 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 7e73a20505744345ae3839dfe5621e41 and timestamp: 1598273973.2899864:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 794
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598273178688/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598273178688/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.061908721923828125 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 795.121s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 17.725 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 18s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/iot6n3v5dr5ae
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1839
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1839/display/redirect>
Changes:
------------------------------------------
[...truncated 290.66 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWfarnlIACKN/////////////////////////////////////////////4AjT6gN26rdTHZtCxsgpPWFROINJCmaKYnpo0mEzUNPSbSepo9TJkbUNBo09QNPUADTTQAAAAAAAaAAAADQMQaNDTQBoAGjTahkwjT1D0mmQ08kGSEaNEZMEI0aFPKPUNMmmQAAANA0aZAAAAAA00AAAAAAAAAAAAAAAAAAAAABSVP2Sj1BDATJoHqGJgEaZMEyaDCMJkyYAEGEwDQmRoxGmhk9CMRkw0NEGmTEaBiDAmE0YmQMmmmBGmhoaGAQep/qkGITACNMTTABMEaZMBGmmTAEwBGACYAIwARppiYATAAAjTAAJgAAmmAAIMIaNAwAJMABMAATAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASJEAjUwhqMjVPym1T9BR6jQeKB6maj1Mmm0jEAANA000AZGmmyamgAAADQ0AAAAAAAAAAAAA0AAZZMS1uZjAmlzAsMvq9MjhRO4Tx2bNrDjwDJaL7IoVi+XJk369d17etfeZTCuZWgG8opdBVVEuh0QhMl/fGsEQgIUKBASNMSbSAZp6J3+aS5VMY13boHqEpsY0JTEIJ1rJIAxZ3cLAUCeHCNAtCAL4MV7ljBovrt23fxXl7czTTwlhpmEvM48b/g0kjxBoBE4FqM0jaGzNQ3WPTDMWXFp0hFgt7Gec6TMB0h4ckFZeF/gCo8TcF9pMqRViJB4Gnb5SoKp0kwmE4JyFKwlMJgTrJ+QmpygI5/wUcToeibc5zkpVcWKk3NnFkZq4Zh5+glOmEXXcO2dWQvCV8BGIlDkokSkNhM0iUkNjeQghsOnEKNsSGS0XNMhtMjEdhGcqTvM82QUGYmWLGGiaZSVkysps60FgQzm4vZyyoOwBIyowfU0O44P/WjCRVqnV2Ac2KtlipHwc/woATKkpDtUFNUi5EfnGamPfw/9b0aLCiZNExl3B99DxZfqzZs51+0UUUUUdDGIQhCGMYhEk5x+FXRQKwCFsAgGYSFGu7ZgW+j4HO6y5ORwi3lBUC6aJpq5hashObJiboZCx8TOoMAMOmlRfLSoxthLAMvzJzBy8BbhmNTzX2Y0vb8uooLdtkaS4SjfHQ7U4i3VE3nDz3etBZ6kZk7h9iTgbOZg2uU2/cAoZMPNGVEDq8aJiYnGDCLpobKLjIicoLYoi6coJ3mFCiApqsUCmY12Q4cHHcRbssU2a8zx5ImcoGG5rvYBE1xBr2zfxPgQBJ0Iyf6MstR4dlf/xOOwyiBYonuI2MqPSgIEbCYMAMEhG4AxtDodTtdvu0eBXuJmOlmZjxTc58p/80sK9gxjAwszy82N8WvpyslZPJ5IdERZZBRBZ6zE70B4AgoJ21BXQB0QZfb7SMY+pVs42AhAOAYaml8ssa1dEjrKeL8846yLCNdZKMVR1ijUm3ZR+5MtvyMCidKGRWnIOR0geJJ6OzWwe3qCwuBLebJN5GcGssXpnFQQIKVWOPsBSZHPoxJIxBjMCIFDsIsyUktkfASYIedh1MszpqKG4i6UCk1x8qg6fTMoK4oldwIwYxWKVRdEnaTnuJ49Qp35UutqphXMeVTaBZObuKijBQDC7mrO+o7KmKxJKtVBeLIY1gF14FaxNm7Us90Aa0GYOQS6FjaZC/TDEtIgyN6cytNQ2jwLYV1yXj6EKjMRAEmJUSBIgeWbo+ImyYA4fmapgnMQcTiDuVol+5b+3C8asZY+R6coe/SqjvGHgqgFh6+dlpeVSFBkZY8wZVqtuBAFBgGcBFKAiAZgAURnyLiJqklIu4RUaR2okMPclIdGeDsNjYVCalilrGOuaEBsLsLYgTHoDAgIFNL5NAOKaxBlECnSGSJxLE3ZvL966DvxcQYuPsVYsS41b9Wqhu6ldYUqlFGBXoFTTZDrqFCqm6JN1VGZJLCv+2W2EzERwCdu/qKp3y44IrDOkaxTpT5gVNwtpTlZS6CVsRMi7QaB0zQRyygmBsmCgzUk7FicYBU8gqLS4e0Cqml+M1zagX1ym/NFIWvSFA3ti2hWLgLVASg4E5WwTgPBYW2ulZ6C8KQtdxVkYktW8TM6nEMDCAKanphpDkHEQwgQmAwQUrYR0vsa4FWAUwzoKKIlmjkQBLFlTdK4hAQ15hLAuS3QlQKigIJQ5j7GIXEBm1oadkFwDIAwFzApZ9o3ugCtMHRqYOzu9sbY+V1cS4iYhYYB9A5G7kDg2ApHItXM5IZXbvlZ185caG2amWWQIxmSwVrNdOnxgMuqVDvFrKERIbU4jjr6jSQdmLHik4tKdZEYcBcxwNjVxj2YDFgqF0ynIQkG1cKGFF4LMqV0YUpXkn2YJv0wFJLVDNQGqZ92lZTiCBVsUXPCO+3gJ6GdWVVw1WJNgllampLPVoMmpgylNu4p0YJixXkWHJlapa4qot4OAWDW2tAVjRnNFNYYAoVxY1NCdJBFEAXuQp+0wzE23cy2ClhJUtWiiHMtTGSaghW74xTnomrBueX0RSv7rrsWzSCNkbBmzFJlrMDCkIFYLKkKQh5RZizzLqSHNygkD05fJDm4SppTCbeyg2BFLVyMrOMWmgWI6JSww2DZrQFKwbWK0q4wxslKvGwGzQLE201UuZ9b1ymWfiIalzkPkuciIogg1Ba15ku31kDHQQQVmQQvPS20SGdJIYYYTDXaxdKnlwb4jER+WWRS0t+Uxlf3JoLjnrD7zGxLvHlPOdbTDSJyhJGoMqtP6e9GVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEERKzlLSmxCKaXOapUC0RERHOUtHOcHOiiifnWI45lvxxFzlkyZ0m2226KMEqpUSbbbtxylw1bOcRGDJKVM0WMWXDI2ecUZmlKaW219WeFNaYmMZqKs1jVgKPtamDdOzeX3gpCVhdt1uMudOy2odcJcawUVjjXVgqcGIg5gNEzGmqhbgpe5HNjUt4uqYSSztLY3h+dvs0H4QumdorAwDI1A6CcZkA1BkTJOZ3phOKlWVyCyl62cvAwDLAsOHLIMj2NfwpQbQZ6UZ2cNzjMRglvWlMJyy7F9GRomgBTR33AhZoRQIgNlWd1S3GCqC4GgqrlxaIBYmmagTFMdaUTxztUEGWivnmv+w5ushET8GKWKnYl2gmis/HSVgIIJTg2hTEIkIt4Htdh183y/X/OpVEYswJoXjp8jSXqgLuE9ajWWnwdJI0ldSSL0bPdqSNOrUJSrXGWGoUmOS2w7nDBtWlFjCVRRebdUU012lSwbxTsN80Nk19NimlBG09dDiH4ZASYMEAvASrwY00zZ9J1+FnXP/uY+75T1mn6H5/S10X1UNFH2+/4apKNIqpP1S037/tP/mzeogo4QIpLGcIsBkACEED4QLwmAMAYIBAJADGAueIeBT/67HgNzohtFNz6hyDZs3uqcGNig7CoRdglEH34/aMkDYNMGwaYNgxfdiA2QOkggUhoBtJDbaB+6gRzTQeWaFWNLV9V/8XckU4UJD2q55SA==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-24T06:46:40.526914Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-23_23_46_39-11870257176693943788'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0824050319'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-24T06:46:40.526914Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-23_23_46_39-11870257176693943788]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-23_23_46_39-11870257176693943788
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-23_23_46_39-11870257176693943788?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-23_23_46_39-11870257176693943788 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:42.877Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:43.577Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:43.613Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:43.777Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:43.804Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:43.883Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:43.931Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:43.962Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:43.997Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.032Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.058Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.081Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.106Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.136Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.172Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.198Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.224Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.262Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.290Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.323Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.356Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.393Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.421Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.464Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.500Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.535Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.564Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.785Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.841Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.875Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.887Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.911Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.911Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.971Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:44.971Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:45.040Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:45.066Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:46:45.147Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:47:10.911Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:47:14.286Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:48:43.364Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:48:43.389Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.567Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.622Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.647Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.716Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.739Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.773Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.780Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.792Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.826Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.832Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.858Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:51:58.892Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:54:13.023Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:54:13.082Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:54:13.125Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:54:13.185Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:49.636Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:49.696Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:49.745Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:49.817Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:51.862Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:51.922Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:51.996Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:52.027Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:52.046Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:52.081Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:52.105Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:52.144Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:52.218Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:54.607Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:54.665Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:54.747Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:54.794Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:54.854Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:54.911Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:57.251Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:57.322Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:57.401Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:57.485Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:56:57.516Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:57:45.550Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:57:45.587Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T06:57:45.623Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-23_23_46_39-11870257176693943788 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598251596715/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06810641288757324 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: ede32645e83143aabef0133125609f4e and timestamp: 1598252388.9067647:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 791
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598251596715/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598251596715/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05318903923034668 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 792.769s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 15.852 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 29s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/jy6vsgyeb6w7o
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1838
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1838/display/redirect>
Changes:
------------------------------------------
[...truncated 290.18 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWTFF7DkACKN/////////////////////////////////////////////4AjT6gN24uCNmyLGyAT1hSTiDRQhkpieI0j1NMR6j0m0nojTI0ZDQDTQNPUAGgBoAAAANAAaAGgAAANADQ0AMgAaB6hoaDTQ0GQ02kGSCDRMRiT0jRojSPUNMmm1A0AADRo0aZAAAANAGjTQAAAAAAANAAAAAAAAAAAAABkjYoAAAAEwCYAEYATIwAAmACYABBgTEYBMJkwAAAAAAAAEwJoyMCDAAACaEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRNAhMJNqMTVPyFP0IIB6agaaeo9TJpiMQAA0A00AZBpsp6mgAAADQ0AAAAAAAAAAAAAAAGWTFVZcxgWJsyK7L2SmVwoncJ47q6qwY78y1F5kUKxerlyb0kjry8feaymFdhWQN3RS6CzZiXQ6IQmW+vTWCIQEKFAgJGmJNpAM1NI75NJcumMa7l0D1iU2MaEpiEE67kUAYs7yFgKBPEBHwKhAF8IJLdfAovbly1fRXd5bznUwlpqGExGgeM/waaR4c2gROBazNM1TbGscVk1Azlt5cdGRaLjo0TniZgOjPDEgsMQYu+lZ4i8MVxMrRXjJB36m7zFYVzoTCYTgnIUrSUwmBOwn4+anKAjn+/xw/B9C25znJSr4kVpubOJIz2QzDz9RKdIRffwrp15TEEsUBGMlDkokSkNhM0yUkNjeUghsOLIUbkkMlpOaZDaZGM66M5cniNE2oVGcmWrIGkahQsJlhS3qoWhDOcjEzmFUdcCRmRg+ntO24P/VyOiq/7G13dot9b4CSleN2UgJdgW+6fvnqIW8kMk4YSD8H7neDL3sS5165WwIxoj2S0xNztDS1IYYYYYsEECCCCCEEECCJE944IRCUBDAFEcAEB79PqCSkxRR+i4HPbC5SRypdywqhdJE01ewuWUnNkxN1MhZOHoVGAGHUSovF8qMb8JYBX5o5k5iAv4kxq35q8YWVcBE11RUGpdnIlWhC2KxczNBM0qNc6SGgzlYslIwmSKKCINsULeU3LcwoZMNfGVEDpI0TBweNGAXWIakXGRE5QWxRFw5QTtYVKICldqgUzIuwHCg5DhrjC1TZrTPHEiZywYb2u8BJHpnovJsPS6DCorEsS6uyikDe+xMfqBTLTr+sx8p1LUKlwNLZEzZhmiSpUDOPj89qpHXp7So7jHcnHy3a9Ex7Zn8TtUegZmDFaZMqu9tXxB9J9JkiSWlSUUNWalEsyNnC0GrJO1FEgkCka7VY+xavm2dVFGv2wsPM4UpqdFSt6yh1sZ35h3koi5u/EuTG61aNVyLeQ1ptbmnAJoco02qCTqhvkRxKQhkNK9UiCjrwWw06nccQjSreGYYhIgwY4NGplxSQfBfyTiTOIIgUPERhrCWVh7BKBLzkOtluhNRU3EXygUmuryrDjKTKiyKpX8CMGQVqlWXxJ3E57yeTWKceVrqyrYWTHmU2gWXnLysqwVAwv5u3vSOwpitSSsVYYhZTIsAutArmJs41S0XUB98GwHIJtGxlMpephiWkQZG2czPmyNr8C0Fdel28hCozEQBKExJIkwO76PvuFMSANv3pxQLTIOiBB5KudHq2TqVTdM8YjJcvlI0yTXdcxK0YCq8/JynRk1ZQhG0ZQhU6nqiAKEANUIpcEwDiACiNGReRNUJSL+VKzTOzEhh7goHRHgbTZ2VUmpY5bBjkNCA1111sQJjkBgQECml4mgHCNYgzCBTpTLE4LmRIdAk0FGjLiIKE2lQmAGY58dHLi1EieOQK3ypsZkIKlLYdlYoVc3VJuuszpJYV/2260mYyOATu4+srnilyARYGhI2CmnPmhUvFq0zMo6iV0RMi/aNA6TQRzCgmBtWCgz0J2rG4wCpyKquLx6oV0o/F7o24F7bpvjRSFr0xQN5XtIVi2C+oCYG/nK2CcBoWFvyJTePBhgGryKUnXiz9Hl/YtEMDD+Xm90tNw5B0EMIAKAXkMGsKvjboqJmClnHvmomUm3UgCaNKKvgUn+Gt0LU2yy0kxBTT/BLnYfQyFmAcTnwn24bANwDAtAE7Mzbm0gPqgXzhh4eXqLsL3UGxNiKLDQwHwDkr+RODaCkck1ezkxll2/VvWzmRobZrZpZQjIZbRVZ1qU+LBmyVkd4tZQiJDVOI45NVpIO0ljSk4qU66EYcBbxoNfWhjmX7FgqFwzHHwkGq2UMKLwGZUroxExXkXmYBvkwFCWuGeoNc0b9O2mMIFYxRe8I8V3AT2mhYV2Q1YJNglmampLRVwMmpgyibd5TSgmLHiItOULFRriKq7g4BYNjb1BYNGg0UsDAFSvLWpoToQRVAGJyFP2eGYm272XQUYSVGrhRDmXJjJNQQrt+Y5z0jXg3vM6QpYu47pi2yQRtTZM+coZrFZNO0MJJQ07SqFPeZsstJ28WmaBVmVEi6vcRlxq03dtRsiKNXozM3C1ECxnQqWGGwbNiAosG3ixKyMMbVSsyMBs2ham2mq13P1fWKS0cZDUudh8nzsRFUEGqLWvErtdZAx0EEFZkELWzWkSGdJYYYYTDI+xdKmrgzCMRHpppVLS15TGSe5YgtubXeax+Jd21TXOtJgpE5QljUGWantTbGVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEERKzlLSmvCKaXOfUqBaIiIjnKWjnODnRRRPTrEcdhb0cRe5ZcuhJtttuqrBKuVUm227skpcJXTnERgyylSaLWLNhkbbQKs7SlNLc7rXnhTWoJjGaysNg14Cr7Otg4p277FiBSErS/dreZtCdt1Y7IS6YwUWDjdK0VMGMg5oNIzmoqxbwo96ObGpb5dPwkloaezvj8zfYoPwBcM74rAwDDsg6KcXjw+gwzLOZ22BOLKrpcoukvH5y0GAZoFhw5pBlezuvCFRqjPSDOxhvchjMEuOaUwnLNs4qstZlsE+BS4RJaLgRYs8b7Sx6szSs1SsUYlcqywyZcr6BMIx1phPHO+oIMtFetfe9hz+uBET8GKWLLsVcoJorPR0lYCCCY4FQpiESEWr/2s3m7Hy/X/OrWSMKwCaF42fDpLbIFzENuRsLU4OmkaavoSMSNtxqkjUr1iUrFuLTWKGSS3I73DBtXFVrCVZViN2qqUsuK1g3ynab9obJr6TFNYCPOVAOgpgwwkwYSEGAsgwZ+Zw45Lv6q22fvPF+f2+6tTPw+PysE18102u+n1+04r+6tj+f1Oz37/s//qBzQwV2EAPzV0BGANwAQggjCiDCgBgGEgkFYBmBaL9RxP/61ogOUImginKNC6igoHNqWgzwAeBQpvASiD70fsGSBsGmDYNMGwYvuRAbUHQggUhoBtJDbaB+5gRzbQeVaFYNLX9T/8XckU4UJAxRew5A=",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-24T00:46:31.108998Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-23_17_46_29-9827459840762801037'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0824000554'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-24T00:46:31.108998Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-23_17_46_29-9827459840762801037]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-23_17_46_29-9827459840762801037
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-23_17_46_29-9827459840762801037?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-23_17_46_29-9827459840762801037 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:34.434Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.297Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.333Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.412Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.438Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.541Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.587Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.622Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.649Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.685Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.716Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.748Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.795Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.830Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.861Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.888Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.922Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.957Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:35.995Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.030Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.054Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.085Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.120Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.192Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.225Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.263Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.296Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.449Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.525Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.549Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.560Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.583Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.597Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.653Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.653Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.712Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.747Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:46:36.795Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:47:02.956Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:47:05.230Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:48:42.319Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:48:42.359Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.528Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.608Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.643Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.701Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.736Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.761Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.792Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.831Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.868Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.888Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.896Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:52:02.956Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:54:21.179Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:54:21.252Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:54:21.310Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:54:21.386Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:01.306Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:01.372Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:01.439Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:01.514Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:03.766Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:04.025Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:04.091Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:04.112Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:04.141Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:04.167Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:04.192Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:04.244Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:04.318Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:06.648Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:06.712Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:06.774Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:06.840Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:06.909Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:06.971Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:09.517Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:09.587Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:09.750Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:09.805Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:09.839Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:58.130Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:58.174Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-24T00:57:58.205Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-23_17_46_29-9827459840762801037 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598229987312/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06815385818481445 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 6f5be9c5d81c45e7b839b1350766d071 and timestamp: 1598230780.3068867:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 792
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598229987312/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598229987312/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05867767333984375 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 793.511s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 16.197 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 24s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/2abm6gmy2ut7g
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1837
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1837/display/redirect>
Changes:
------------------------------------------
[...truncated 290.83 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWWYnPLEACKN/////////////////////////////////////////////4AjT6gN24rhjZtCxsgpPYNROINFCGSEemjSPU0xDT0m0nqaNMjRkNANNA09Q0A0AAaAAADQAGgBoAAADEGjQ0AMgAaB6hppoNNDQZDTaQZIRoEZMEIyaNRqPUYmmmQAAAPUaNGmQABoAABo9TQAAAAAAAAAAAAAAaAAAAAABkjYoAAAAEwCYAEYATIwAAmACYABBgTEYBMJkwAAAAAAAAEwJoyMCDAAACaEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRNAhMIIyNU/KZT9CR6jQfooHqZqPUyaYjEAANANNAGQabKepoAAAA0NAAAAAAAAAAAAAAABl0wbG6mICWTEFZlurRHAJ5oBNFXr2MXFfmOwXWQvrFtceUdq1YL29a64yh9cqtEN5PRA9TTCueCEPmO+tmoEQgIUKBASNMSbSAZqaJ3yaS5dMY13LoHqkpsY0JTEIJ13DQBizvIWAoE8QEaBYDwXwgq2q2BPbuXLN9DeXtrOdTCWmoYTEZx4z/BppHhzQCJwLVZpm2NqapxuTUDSLby46Mi0W+jPOeJmA6M8MSCwxBi76VniLwxXEytFeMkHfqbrMVhXOhMJhOCchStJTCYE7Cfj5qcoCOf7/HE8H0LbnOclKvixWm5s4sjSshmHn6iU6Qi+/h3TrymIJYoCMZKHJRIlIbCZpkpIbG8pBDYdKIUbgkMlouaZDaZGM66M5cniM82gVGkTLVkDRNQoWEywpb1YLQhnORiZzCqOuBIzIwfT0O24P/WG/w9k9rS90iIemsZIS3I7SQEwvLhdT30DgXAjss2Xxz4Hzn+HN4ATB2OLuARuwj2S0xNztDS1IYYYYYsEECCCCCEEECCJE944IRCUBDAFEcAEB79PqCSkxRR+i4PPa65SRypdywqhdJE01ewuWUnNkxN1MhZOJnVF4F/USoulpUYmvlgFzmjiHMvFuKNNRbiscV8BmmWCtQvEvThlWghy2MQsfbUYx3NKnSoaDKViuUqBMmUUEQbWoW8ruO5BQygcaMsHoKsSJg4PGi6YJX7KLiIhgUFsURfHKiZxhUogKV2qBTMi7AcODj+It6WqbNtM8cSJnLBhva7wGL8VOPzfP+9EN0W1m8VKr7+OzHlfD+mMePK1vIo9lMjyzkgRnQm7dF1ZcVAzjY3Ra6P2ae4q28yXLycv2/TMzU3+J+su4MzBivsWNceGbyg9E9ExRJLUpKKGrNRwtaLULQask4ooYEgUfYq1G2fX86JVQ5ue6Fm5vGldbpqdwWMItifBMukhCxbpIxbC9VuLVx7WO2Js7um/JoSZps0EnJDhIiRpiGU1LvTH+/Xwtpq1O62gWVb8UywCRBewxCNYKSJB5GEJOGM4UiBQkLEmsJZXXgEoEvOQ62W501FTcRfKBSa5CVYdLpMqLIqlfwYwZBWqVZfEncTnu55NUpwCtdXVbCyY8ym0Cy85eVlWCoGF/N295R2FMVqSVirDELKZFgF1oFcxNm9Us91Aa0GUQIJNGzyiMu0PwrSEMicmMjTSMx4NkK69Lx1CFRGEgCUJgSRJgd70/feKY0AbvvzygWmQcUCDsVh6XWtPVrO+qGQRkOZJovcou7BiWpQFa6GXlelKKygSNkxgSqVTVEAUCAaoQ64JgHCgFEZ8i8iaoSkX8qVmmdmJDD3BQOiPA2mxsKpNSxy1zFVM6A1l1lsQJkEBeQECih0mcHCNQgyB5UEZjhgJY27dnFvnUd7LyDHyFqsFjXHLgK5VN30XVFKtRVgWJAqUth2VihVzdUm66zSSSwr/tt1pMxkcEndwKyueKXHhFgZ0jXKac+aFS8W2pmZR1EroiZF+g0DpNBHMKCYG0YKDSoTtWNxgFTkVVcXj2wV0o/F7k24Fu1RfGegLXpigb2tZQrFoFqgJAb+YrYJgHAsLbVSo8mAC8M3YU5Oxlo6XMe1aIMGH0vP7pedhCDiIMPwUAvoXs4Jzu14VFUAUu2985DSs65EATJlSXOqJPsHcoKsNFluJgCnH1/XUgfQyFrAbT3woWoaANQDAtoE/Nzre3APSgc54w6uzvEWZ4qrQmhFEBkYDyByV/InCtBSOSavZyYyy7fq3rZzI0Ns1c0soRkMtorGdalNjQZMnSO9WsoRChsTCKKrqtBB2kscKTCwqCuIhABax4NbWhkGX7FgqFwyHIvkGxaKF9F4DMsVz3ZCvDdZgG+TAUJawaVQaxn36dtMYQKxii94R4ruCnoZ1hXZDVgk2CWZqaks9XAyamDKJt3lNGCYseIi05QsVGuKqruFgFg19vUFg0ZzRSwMAVK8tamhOhBFUAYnIU/Z4ZibbvZdBRhJUauFEOZcmMk1BCu35jnPRNaDeczoili7juWLapBG0Ng0tIoZrC8wo+BUCypCj4cUWYU0q6UfxhQRhyYukiBt1UshfN1bUbAijV6MzOMWogWM6FSww2DZrwFFg28WJWRhjaKVmRgNmgWptpqtdz9Z1iks/GQ1LnYfJ87ERU89qi1rxS7PWvMgeeeqMefccksoj8yRvvvvl+q1i6FOLezKMRHZJI1LSz5bGVfcletQOVnXGNhXeOKcggspi0hgUI4lBlNh3U3IioeHiYQBgwlKxQAAAAAAAAAACwABoABGLFkwoaBAAgggggjeG6JPQyWmhoXXVxFkm227rok7rri66+++maRgwZZUwRF7lly50m2226qsEq5VSbbbuySlw1dOcRGDLKVJotYs2GRtc4q0mlKaW43OtPCmtQTGM1VYa5rQFX2dXBxrt2cWIFIStL90t3mzp23VjshLjmCiwcblWipgxkHNBomkairFuyj3g5salsrqGEks7T2Nk/M32KD8AXDM0VAXhkqQdFONyIagyRjmMzkomFKq64xXS9bMXAXhmgWHDmkGV7G58IVG2GekGdjDeZDGYJb5pTCcs2xiqytErwKaW7ACFmdFAh411Z/TJaYKXrQaCqqYC0PCtLK1AmEYqkgmimaoIMvDbca7m4G64ERPwYpYpgwbk5M9R2KgrAQPSHAsCiEQoRZv/a7HsJfl+v+dWpIwpQTOvHzZKguUgXLrlyNdanC00jTV9CRiRtd6pI1K9UlKxcZaapQySW4He4YNq4qtYSrKsRulVSllxWsGyp2m/aGya+kxTWAj0FVDiHsYoSQMJCAAWQAM/O2bct09dgaPnoiPRNe6vzfw+Pyss58106u+n1+08r+6th+f1P0H7/tD/qJvRv96D8PzXEBEgNQAQfwjBCACgBgGEgkFYBmBbcHSxQ/6KRAboQ1FDt0aNyFFRN7etBnGB1FGm6hKIPvR+wZIGwaYNg0wbBi+5EBtAdCCBSGgG0kNtoH7mBHNtB5VoVg0tb1P/xdyRThQkGYnPLE",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-23T18:46:32.889978Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-23_11_46_31-8898380291330454735'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0823150256'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-23T18:46:32.889978Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-23_11_46_31-8898380291330454735]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-23_11_46_31-8898380291330454735
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-23_11_46_31-8898380291330454735?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-23_11_46_31-8898380291330454735 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:36.063Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:36.927Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:36.970Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.032Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.061Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.165Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.209Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.244Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.279Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.311Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.342Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.372Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.406Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.431Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.466Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.502Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.536Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.592Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.627Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.661Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.685Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.721Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.751Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.787Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.819Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.853Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:37.887Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.042Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.110Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.143Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.155Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.179Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.190Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.248Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.248Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.317Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.356Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:46:38.391Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:47:05.048Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:47:08.138Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:48:39.545Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:48:39.594Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.602Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.682Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.706Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.779Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.804Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.824Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.839Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.864Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.892Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.905Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.943Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:51:57.977Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:54:21.800Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:54:21.888Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:54:21.949Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:54:22.022Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:56:59.788Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:56:59.860Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:56:59.926Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:56:59.996Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:02.339Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:02.408Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:02.480Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:02.516Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:02.529Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:02.567Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:02.621Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:02.646Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:02.715Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:04.873Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:04.939Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:05.024Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:05.077Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:05.152Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:05.220Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:07.542Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:07.630Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:07.712Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:07.837Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:57:07.876Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:58:03.403Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:58:03.461Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T18:58:03.497Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-23_11_46_31-8898380291330454735 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598208389286/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06461787223815918 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: ad7d0bc79ac4469391859c4ab63fe48a and timestamp: 1598209206.1238163:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 816
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598208389286/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598208389286/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06682324409484863 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 817.407s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 40.606 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 49s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/tqoeqeodmqml6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1836
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1836/display/redirect>
Changes:
------------------------------------------
[...truncated 291.13 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWchwHzQACKN/////////////////////////////////////////////4AjT6gN2cudMdm0LGyAdvXConiDRQhlJiemjSNNMR6j0m0nqeo0yNNNDQDTQNPRAGmgaAaAAAAABoAAANAA0AA00AyABoHqGJpo00NBoNNpBkiBPTUxME0mJok9I9Q0aabUAAAD1GjRpkAAAAABo0AAAAAAAAAAAAAAAAAAAAAGNTxJ5SNBDI9EaYg0NNDTTINpNMmj1HpBiGmTCZMhpgRpgRiBpoxDCZNHqaGJtIM0QNBppoBpoyAaaGBGCAaNDRkyaEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRNAhMJMjI1T2jVP0InqNB4oHqZqPU2ppiMQAA0A00DRkHqbKepoAAAA0NAAAAAAAAAAAAAAABlkxtW5loEsm/Cw5diojYJ5mCZtmzVhNwDHUXnIayxdXHk3oomW7b7zrlFZcqtBN9PQyCmmFc7IRWMd/dNcIhAQoUCAkaYk2kAzT0Dvk0lzCYxrufQvUJTYxoSmIQTrWKgDizu4WAoE8OEfAqEAXwYiuWMGe7evWr+G+t3M008JaaZhMRnHjP8GkkeHNgETgWozSNmbI1DjsmmGiW3lx0hFot7GedATMB0h4YkFhiDF30rPEXhiuJlaK8ZIO/U3GYrCudCYTCcE5ClaSmEwJ2E/HzU5QEcPv8cXwfRNuc5yUq+NFabmzjSNGyGYeHUSnSEX38S6deUxBLFARjJQ5KJEpDYTNIlJDY3lIIbDjyFG3JDJaDmmQ2mRjOvDOYJ4jPNiFRoky1ZA0DTKFhMsKW9XC0IZzsYmcyqjroSMyMH09h27B/6w3yHskyzvLnEPjWPkpXkdtICXXFwOr7/1SC30hk2y6QfQ+9HxZm+iXOx3W3hGMNiU0zxptPylBxxxxx8YxCEIQxjEIIb9CblVkxmMAjMEZCw0ONpakiPo+B0GquVkcsXcuKoXSxNNXsLllJzZMTdTIWTi51SsFbwlCoq1CwV2qYZvNriLh2KvwWssMquLLpynEtFQaTacUqz4PC2GnDdzCq0jmYSZIjOZisUlIgmSKKCINpULmU2/cAoZMOvjKiBkTUTCwuMGCWS1rSLaRCxQW4oi8coJnXCpRAUrtUCmZF2E4kHI8VbstU2aszxxImcuGG9rvISR6p6TzbP1uIwyKxLFOTmoo0vvamPzU0y1Lp2F/nOC1SpaDS2pMGAYIkqVAzzGY0Gs12yT29Wkcfy8hLdz1TE1N/egrLyDMwYr7w8Vx6ZPaD4T4TwiSW/EUoDgFUtEFuMQGAONJpjWkw08amzeW3CXhUzgNIBXTmZ6l5vZmJJdoEWYcInwzDrJQsa68a6L7vCRyvXNZDYGyjU39NDmGmyQSckOAiONTEMjp3mmNL9ei2epU7zaBYwnjmF4SILl+FRrBcckHsYEk4YzhSIFW4WJNYSyuvQJQJech1stzpqKm4i+UCk1yUqw6ZSZUWRVK/gRgyCtUqy+JO4nPczyahTfla6wq2Fkx5lNoFl528rKsFQML+ct55HYkxWpJWKsMQspkWAXWwVzE2btSz3UB98GUMQSaHnNEZeorQrSEONdmMj5pG0eBaCuuS+eQhTTCQBKEvJIkwO/6vtIqYoAbrwT6gcDIOKBB3KwynXtPWrO8YY1GS5vMRqHPW9kxK0oCtdLKScpz1ZQJGxeIEqlU1RAFAgGqEOtCYBwoBRGfIvImqEpF/LFZpHZyQw9wUDozwNpr66qTUsctUtiM6A2F2FuIEyCAwICBRQ8TODhmuQZBApkZjhYSxt27zFvXUc8XkGPkrVYLGuQW/VyqbvouqqVairAsSBUpbDsrFCrm6pN11miklhX/bbrSZjI4BO7f1lc8UuRCLAzpGqU0p82Kl4tnTMyjqJXREyL9g0DpNBHMqCYGxYKDRoTtWNxgFTk1VcXj2YV0o/G6xtALtyi/M9AWvSVA27FpCsXAX1ASA4ExW4JgHQsLfiSo8mAC4MncU5OxloaXLzLgIMGH4vR7paerZBxEGH8KARwXMoJ0jLsqJgClm3vnYaTnnIgCYsaS6VRJ+g7jBVhmsiyXgp1+aLXMfMyFrAbT85QtQzAagGBFAUE3PN4sB8UDpPmHZ3eYizPVVZkzIogMTAewOFfyZwbQUjhNXs5QZZdvlb1w5oaG2amaWUIyGW0VWa6fNjgZMlSO8WsoRChqmDWxalQQdmLjpSYVKZZDQwC5jwbGrjIOYDiwVC8ZDkaxBquFDWReC5lSufGSFeK85gm/TAUJbUNGoNqZ9+lbTGECsYoveEeK7gJ7DOsK7IasEmwSzNTUlnq4GTUwZRNu8poQTFjxEWnKlio1xlVdwcAsGrtKgsGjOaKWBgCpXlrU0J0IIqgDE5Cn7PDMTbd7LoKMJKjVwohzLkxkmoIV2+Mc56BtYN1zWgKWLuWsxbJII2JrmjolDNYrJp2hhJKGnaVQp7nNlloO3j0zOKsyokXV7aMuNWm4tqNcRRq9GZm2WmgWM6JSww2DZqwFFg2kWJWRhjYqVmRgNmwLU201Wu6es6zSWfjIalz8PlOfiIqIRKEIcQFEoIJAQhEIiFGNWRAAXAgVSlKU0qofEFYPDCTZgEAD0kkalpa8pxyL3JYLjHbDzrj8K751TrGWkwkhYoRtUHKantQdaVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEERK7FLSixWFFDGPqVAtERERjFLRjGBjIYYXplhrZVvNiL3LLlzpNttt1VYJVyqk223dklLiK6c4iMGWUqTRaxZsMjZZxVotKU0tvrbWeFNaYmMZqKw1TawFX2dTBxzt3mLECkJWl+4W5zZ07bqx2QlyDBRYONZWipgxkHNhoGiaarFuSj3Q5salvF1HCSWdpa+8PzN9jg/AF4zPiuDAMSkHQjjsiH0GIY5jM7KJhSqyuMWUtvzF0GAZoFhw5pBlevreEKjZjPSDOyBushjMEt60phOWbXxVZXxLACmkPMBCzOigRAbKs8pkuOCmC4HwVRFhfEAsSyvoEwy2vIJmzPqCDLQ3XX3vYY/rIRE/BxSxSzG3pyZ67zaCsBBBIcGoUQiFCLWB7XYdfL8v1/zqdJGHKCZ14+bEoLtIF7GO3I1Vp8HSSNJX0JGJGy3akjTr1CUrFtrTUKGSS2473DBtXFVrCVZViNwqqUsuK1g3inab5obJr6TFNYCPPVUOIqAw444MJCAAWQAM/Q2bct19lgZvvpiPTNe6vzc58PjZZ35LZ5b8/p9Z9X9la/8fmg/X7/tD/qJvRtLwH8fiuICJAagAg0CMEIAKAGAYSCQVgGYEVgKWKH/W5EBuhDUUO3Ro3IUVE3trgDOPDsKNN2CUQfej9gyQNg0wbBpg2DF9yIDYg6EECkNANpIbbQP3MCOcaDyrQrBpbX1P/xdyRThQkMhwHzQA==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-23T12:46:30.457160Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-23_05_46_29-7094010668391870518'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0823100313'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-23T12:46:30.457160Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-23_05_46_29-7094010668391870518]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-23_05_46_29-7094010668391870518
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-23_05_46_29-7094010668391870518?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-23_05_46_29-7094010668391870518 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:33.285Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.367Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.406Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.508Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.543Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.646Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.705Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.750Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.784Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.812Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.840Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.873Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.907Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.928Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.957Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:34.991Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.025Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.063Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.096Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.129Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.164Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.187Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.222Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.260Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.293Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.325Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.359Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.641Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.711Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.745Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.757Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.791Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.815Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.868Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.885Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.950Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:35.975Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:36.045Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:46:51.662Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:47:05.593Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:47:05.634Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:47:10.895Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:48:44.565Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:48:44.596Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.107Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.194Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.226Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.308Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.341Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.375Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.405Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.433Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.439Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.478Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.510Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:52:03.541Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:54:20.777Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:54:20.920Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:54:21.002Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:54:21.170Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:00.726Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:00.810Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:00.873Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:00.953Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:03.580Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:03.652Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:03.720Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:03.752Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:03.767Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:03.834Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:03.844Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:03.905Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:03.989Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:06.543Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:06.616Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:06.700Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:06.755Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:06.824Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:06.884Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:09.505Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:09.597Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:09.739Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:09.872Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:57:09.907Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:58:02.164Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:58:02.210Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T12:58:02.244Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-23_05_46_29-7094010668391870518 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598186786830/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06090593338012695 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: e3715b3f2b5848dbabc406c0377963d6 and timestamp: 1598187578.1807082:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 791
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598186786830/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598186786830/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06565999984741211 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 791.833s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 14.602 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 21s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/q2sb3clzd7ji4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1835
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1835/display/redirect>
Changes:
------------------------------------------
[...truncated 290.96 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWcmAn38ACKN/////////////////////////////////////////////4AjT6gN2OudMdm0LGyAdvXConiDRQhlJiemjSNNMR6j0m0nqek9TJk00NANNA09IAABoAAAAAAGgAAAA0DINAAAaABoHqHqaaDTQ0GQ02kGSIaT0TEwTSYmmpPSPUNGmm1AAAA0DRpkAAAAABoAAAAAAAAAAAAAAAAAAAAABjU8SeUjQQyPRGmINDTQ00yDaTTJo9R6QYhpkwmTIaYEaYEYgaaMQwmTR6mhibSDNEDQaaaAaaMgGmhgRggGjQ0ZMmhB6n+qQYhMAI0xNMAEwRpkwEaaZMATAEYAJgAjABGmmJgBMAACNMAAmAACaYAAgwho0DAAkwAEwABMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABIkQCNTCTI2k1T8jVP0InqNB4oHqZqPU2pptIxAADQNNNA0ZBpsiaAAAANDQAAAAAAAAAAAAAAAZZMbVuRhAlk30WHLsVEbRPM0TQ2bNWHDhGOovOQVli6uPJvRRNt22POuUVlyqz84E9DX6aYFztgFYx4N01wiEBChQICRpiTaQDM2gd9mkuVTGNd16B6hKbGNCUxCCdZySAOLO7BYCgTw4RgFQfC+DEVyxfnu3r1rBgwLdzMs2EsMxhLzOPGf4NJI8ObIInAtRmkaZtDUOMx5g0Sy4tOlIsFvYzzniZgOlPDEgrLwv78VHiLgvtJlSKsRIO/07jKVBVOkmEwnBOQpWEphMCdZPx81OUBHP+Aji9D0bbnOclKrooqTc2dFI0a4Zh5+glOmEXXcS2dWQvCV8BGIlDkokSkNhM0iUkNjeQghsONIUbckMloOaZDaZGI7AM5UneZ5swoNEmWLGGgZikrJlZTZ1kLAhnNxezllQdfCRlRg+nsu4YP/VuQjK97Wd2d42AbX+T7PF7SQEuuKS6fvnqIW9kci3XSMCIHm+HL3oS51242oI3A2JTTPGm0/KUHHHHHHxjEIQhDGMQghv0JuVWTGYwCMwRkLDQ42lqSI+k4HPay5ORwy3lBUC6aJpq5hashObJiboZCx8XOoVYrOEoVKqULBVYphxPNrirwdaq11r4ZUV8WfUFRGimVaUZlw1G+L6NM4iw019DTK95Xes9ZVI0E7x9hTgbOZg08pt23hQyYdYMqH2xQomHh8YL5bLWtIuEiBqgtxRF45QTOuFCiApqsUCmY12M4kHH8VbssU2dMmeOJEzlAw3Nd6CSPTPReXY+p1GERWJYh2d1FGl67Dj9U9MtO8dZf5jhtQqWg0tmTBgGCJKlQM8tls9qtZsE9rUdzjZXHS3b9LiYmfxOw76DMwYrL4+VZ+ZP6EAUAT4iSW/EUoDgFUtEFuMQGAONJpjWkw08amzeW3CXhU2jGkArpzM9S83szEku0CLMOkd4HDzJxVyeZS3L7pE3hXrG0jrjZXJODTQ5BpskEnZCSREpSEMfpX2kNJC7lsdOp3W8KxifE4XhIguX4NGplxCQfxfiTizOKIgVVio41hLKu/AlAl5yHUyzOmoobiLpQKTXISqDp1MygriiV3AjBjFYpVF0SdpOe5nj1CnflS60qmFcx5VNoFk5u4qKMFAMLuas5xHZExWJJVqoLxZDGsAuugrWJs3alnugDGAyhqCTQc3ojL1FaBaQByF2YyMNI2fwLQV1yYDyEKhMBAEoS8kiTA73p++5UxAA3HfnFA6GQdUCD2Vb6PVsfUh92wxaMnyeQjTOWt65js0YCqc7Jc/o8tWUKRsXyFKnU9UQBQoBqhGLQmAcUAURnyLiJqklIu4ZUaR2kkMPcFIdIeCsNjYVCalilrGGIzoDYXYW4gTIIC+gIFFDxM4OIa5BkD6mxmOBpLE3ZvL966DnC4gxchYqxYlxy36tVDd1K6upVKKMCvQKmmyHXUKFVN0SbqqNFJLCv+2W2EzERwCdu/qKp3y48IrDOkaxTpT5gVNwtOnKyl0ErYiZF2yaB0zQRyygmBs2Cg0aSdixOMAqeRVFpcPTCqml+L1zagXblGCZ6AtejqBt2LSFYuAsUBIDhTFbgmAdCwtkSU3jNQuDJ7FKTrpZ+jy/tdBEAxBF5/dLTdVIOoiBBhQC8Bcyhni4XNUTAFLN/fNRfPm3YgCYsaK8VBKCiLZDVNmss5LwU1BNFruPoZCygN5z4T7YMwGwBgR4E7MzbmzgQCgeJww9Pb7G2B+hGZMyKNDEwH8DhXcicGwFI4TVzOSGV275WdeOXGhtmpllkCMZksFVmemzY4GTU6R3i1lCIENUwhhi1Cgg7KXHSkwqU2yIQ0C5jwbGrDIOYTiwVC8ZDkaxBquFDWRd9zKlc+MkK8k85fOCmApJaoaNAapn3aVlOIIFWxRc8I77eAnss6sqrhqsSbBLK1NSWerQZNTBlKbdxToQTFivIsOTK1S10Sot4OAWDW2tAVjRnNFNYYAoVxY1NCdJBFEAXuQp+zwzE23cy2ClhJUtWiiHMtTGSaghW74xTnoGrBuuX0BSv7prsW0SCNmbBo6JSZa1XNOwMJJQ07CiFPc5cktB2camZxRlVKRbVtoyYlYbiyg2BFLVyMrNssyBYjo1LDDYNmtAUrBtYrSrjDGzUq8bAbNkWJtpqpcz6zrdMs/EQ1LnYfJc7ERQQiUIQ4gKJQQSAhCIREKMasiAAuBAqlKUppVQ+IKweGEmzAIAHpJI1LS15TjkXuSv3Gu2HnXGQLwHVOtbaTDSBqhHCoOU1Pac7CVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEERK7VLSixWFFDWsUp9aIiIjWqWjWtDWwQQPTLEMMq3oYi5yyZM6TbbbdFGCVUqJNtt245S4itnOIjBklKmaLGLLhkbTOKNFpSmlt9fVnhTWYTGM1FWaxqwFH2dTBxjs3l94KQlYXbhbnLnTstqHXCXHMFFY411YKnBiIOYDQNEzKoW5KXuhzY1LeLqeEks7S2N4fmb7LB+ALxmYK4L4xaQdAOOyIYgxTHMZnZRMKVWVxiyltkxdBfGWBYcOWQZHsa/hCg0xnpBnZg3WMxGCW9aUwnLLsX0ZGCV8FNGeaCFmdFAh82VZ1TJccFL9wMBVEWlgfFiWViBMQw15BNDMxQQZaC66x72Gs1gIifg4pYpbjb05M9d6GgrAQPyG/UKIBAhFrC9rsOvl+X6/51GkjElBM68fNi0F2kC9jHbUayzcHSSNJXUki9G03akjNVqEpVrbWGoUmOS247nDBtWlFjCVRRebhUU012lSwbxTsN80Nk19JimsBHnIQOopgwo4gMJBqAsagz87dvKvPrrTOB9Eb6Jj3VmZ+Hx+Vfmvmtm1v0+v2nFf3Vr/z+p2e/f9n/9QOaFpdRBj81ZARwDYAEGgRhg1CgBgGEgkFYBmBH32jif/1pRAcoRdBGOUaF2FBQObQ6Azv4ehQpvQSiD70fsGSBsGmDYNMGwYvuRAbMHSQQKQ0A2khttA/cwI5poPKtCrGlq+p/+LuSKcKEhkwE+/g",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-23T06:46:31.524262Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-22_23_46_30-10025249199612656513'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0823050254'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-23T06:46:31.524262Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-22_23_46_30-10025249199612656513]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-22_23_46_30-10025249199612656513
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-22_23_46_30-10025249199612656513?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-22_23_46_30-10025249199612656513 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:34.016Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:34.753Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:34.800Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:34.937Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:34.963Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.073Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.126Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.226Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.260Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.299Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.333Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.379Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.414Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.441Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.473Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.510Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.556Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.591Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.626Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.662Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.695Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.733Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.768Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.811Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.851Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.889Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:35.925Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.096Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.162Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.188Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.214Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.226Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.251Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.310Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.310Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.384Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.424Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:46:36.496Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:47:03.744Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:47:09.879Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:48:34.189Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:48:34.227Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.517Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.593Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.632Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.704Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.749Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.755Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.774Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.803Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.809Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.831Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.883Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:51:49.909Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:54:02.664Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:54:02.744Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:54:02.797Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:54:02.867Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:42.984Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:43.058Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:43.108Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:43.184Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:46.447Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:46.514Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:46.573Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:46.611Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:46.636Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:46.672Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:46.719Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:46.757Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:46.838Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:49.244Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:49.316Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:49.393Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:49.461Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:49.528Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:49.597Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:51.936Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:52.002Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:52.074Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:52.197Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:56:52.227Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:57:43.784Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:57:43.832Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T06:57:43.873Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-22_23_46_30-10025249199612656513 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598165187608/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.07057738304138184 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: fd0e59dad17d4cbb9a1bd6c390b94f36 and timestamp: 1598165989.3078625:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 801
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598165187608/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598165187608/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05904388427734375 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 802.230s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 25.367 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 30s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/evvbwych25lpq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1834
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1834/display/redirect?page=changes>
Changes:
[noreply] Merge pull request #12653 from [BEAM-10758] Centralized Glob Translation
------------------------------------------
[...truncated 290.76 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWaAJMUUACKN/////////////////////////////////////////////4AjT6gN26q4Y2zQsbIBPYNROINJCmZKYmyNJtTTNQPSbSepo0aNGQ9IDT1A9T0QBpoBoAAAAHqABoaBkADQAGgaAABoAGg9EZMQ09QyGQxpBkhGgTEYQjJoU8o9Q0ep6m1AAAA0Bo9T1AAAANDQaNAAAAAAAANAAAAAAA0AAAAAAGSNigAAAATAJgARgBMjAACYAJgAEGBMRgEwmTAAAAAAAAATAmjIwIMAAAJoQep/qkGITACNMTTABMEaZMBGmmTAEwBGACYAIwARppiYATAAAjTAAJgAAmmAAIMIaNAwAJMABMAATAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASJEAgCGoyNU/INCanqNHqeKB6jyj1NqaZPU9EaAA0A00AZBo2KaAAAAA0AAADQAAADQAAAAAABjpg2MoYgK0uXFVluOiSATzQCaKvXsYEV6ZLBdZC+sW1ya07HHBd3bXXGUPrrK0M3U9ED1NMK54IQ+ZLy2agRCAhQoEBI0xJtIBmloHfZpLipjGu5+EemSmxjQlMQgnXskgDFnKhYCgTxIRoFgPBfCiO1Vvp7dy5ZvIbq7tZtpXyw0i+YDCeN/waKR4gzwicC02aJsDXmmclj0gzFlxadGRYLexnHPky8dGeHJBWYAwd+KjxNwYLSZUirESDv9O2ylQVTpJhMJwTkKVhKYTAnWT8hNTlAR0HgI43H6JtznOSlVyIqTc2ciRmrhl/oKCU6YRdd0Ns6shgCWCAjEShyUSJSGwmaJKSGxvIQQ2G5IUbMkMloOaZDaZGI4gzik8BnGuCgzEyxYw0DSKSsmVlNnVwsCGc7GBnMKg68EjKi90uf269/6zxsRapphf3VktaYjk9/l+BACWVFw+zO0VOLgx2al1Uc/B+6fna4MSxrTe7qD4sR7JaYm52hpakMMMMMWCCBBBBBCCCBBEie8cEIhKAhgCiOACA9+n1BJSYoo/oODz+quWkcuW9fFQLaiaauYWrITmyYm6GQsfGw0F4F/UCoulpUYmvlgE3NnLnLPFu/4BqL4B1eGWOrSuVWeB0pcaIwriKOqGmUqVT448531nrKpGZO0fFTgbO6wbHG3TdQoZMONGOHoI4kTBweOF+YKz9KLiIhgUFsUReHGEzjChRAU1WKBTMa5o40HA463RYps2szx5ImcQL9zXegxfipyHs9B70Qyi2sytSN9/Ott8r4f0zl48tV8ijMpicu5KEZ0Rv78X6y45AxmrXT7GQ2qO6rkll+dmJXwtpn4zn5obA9gxjAwh1iyxvi99OWktJYeSEegkkZMZKnMz2gUAyYkboTEEcUjaLC1tsh0mblEGU7wWzocfq7GOqfCWQqhn5pl35MNGO8nFq79YoxNItNXtDL4xGAROlDIrzkHQ7hniTqJ2Zj3uowMbgC2+vS8cvXl1i9EyrCBBUrxh9fLlEc+jDkixMaGIgULDMzJyT2R8BJBDz0OplmGaihuIulApNcCVQdNpmUFcUSu4MXsYrFKouiTtJz288emU78qXWFUwrmPKptAsnO3FRReoBhdzlne0c0mKxJKtVBgFkMavC64CtYmzdKWc6ANaDWECCXRc7okLtD8K0hDInJjK00jbPBshWYS6dQhURhIAkiVkgSIHkbUG+SyIAkvLPpBQYg5nEHkrP1+3b+1YOBVcofJ8+Uls/e4vks+XmDqfwON8H4fvqVu09rr92ttuaEC3YOg4eIrB9WBRGdIuImqSUi7lyo0Ts5IYdIUh6E8FYa2sqE1LFLVMUZnQGquqtiBMRAXkBAoodJnBwjUIMoeVBIZIYCWJuzeYN66DvhcQYuBYqxYlya36tVDd1K6qpVKKLywIFTTZDrqFCqm6JN1VGZJK+v+2W2EzERwSdu/qKp4JdZCKwwyNUp0Z82Km4WwpyspdBK2ImRdntA6ZoI5hQTA1zBQZqSdixOLwqeUVFpcPYBVTS/GbQ2IFu1ReGegLXpagburZQrFoFqgJQb2YrYJgHAsLbGlX5sEFQXvIqaNpLUPUtNKBCAwfy9s3Kt4Ug5iEEAEgMCFS+DUxd8ciqoKVl52eY9Zu6EAS5dT1NaQf4S7wdfYJ4olYKef4F2dR9jELiBLz/yo2gYANADARIFDON3EUAtSCmfMHh5e2VsfK2wJgRMguMA+gcO7lDhWApHDauZyoyu3fKzrpzI0Ns08ssgRjMlgrGb6jNkAZdXpHfLWUIhQ2JhFFHqlBB2oscKTCwqCuIhABawwauTGIy9YsFQuGU5F8g2LRQvou+ZrpXPfyleSdZfG8S8UktQM1AahnXaNlOIIFWxRc748FvBTz8NZVXDVYk2CWVqaks5WgyamDKU27inQgmLFgIsOWK1S1yFRbwrwr2rsaArGjC0U1heChXFjU0J0kEUQBgchT9tfmJtu5lsFLCSpatFEOZamMk1BCt3xinPQNSDcczoClg7ltGLXpBGuNYzZiky1quadgXyShp2FEKe3y5JaDs3KZhKMqpSLatlGTErDbWUGsIpauRlZslpIFiOiUr8Ng2asBSr2xitKuL8a5SrxsBszyxNtNVLunrutUyzsRDUueh8rz0RFDz2prWvNF2eueZA889UY8+45LZRH5kkffffL8bWLoU4t7LIxEdllkUtLPmMZH7lZ61A5VdcY2Fd04pyCCymAkMChJEoMpsO6k5EVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEEbv2xJ5+Owz8+22qIrk223bbEnbbaW23XXU5ZF69klTeiLnLJkwybbbboovSqlRJttu3HKXQq2c4iL2SUqZosYst+Rr8JRmaUppbPaak76a0hMYzTVZqmpAUfZ073JOzeYMAKQlYXbZbfLhnZbUOuEuTYKKxxtFYKm9iIObDQMxpKoW3KXuBzY1LeLd3ySw6OtvD8zfY4PwBcMzRUBeGLSDohyGRDUGKZJjM5WEwpVXXIK6XbZi4C8MsCv38sgyPW2nhSg2Az0wzsgbjGYi9LetKYTll1sFGRorPAppTsAIWZ0UCHjXVn1MtpgpetBoKozAWh4Va1ZqBMIxVJRNFM1QQY8NtxruZgbjBET8GKWKYMG5OTPUdioKwED0pvrAohEKEWb32+xzFb5fr/nVaSMKsCZ14c2LQXKQLl+5ajVWlwtFI0VdSSMCNfulJGlVpkpVrZWGmUmOS2Y7nDBtWlFjCVRRgNsqKaa7SpXt4p2G+aGya6VimnBG0lbDmKsMcOkDBAQQCeCBje6YmJR399mYPzZk2+M3Dzny+f0tc99XZu7fb7/ifTUCZX+v3Q0X8/1HSUrimgb8IAfuxnBmA0ABCBB8GIIJAGAMEAgEwBjAROGehR0l0PAcHMaWIcH0zoHDimuagGNiA8D3ovASiD70fsGSBsGmDYNMGwYvuRAa4HSQQKQ0A2khttA+kgRzjQeWaFWNLU9X/8XckU4UJCgCTFFA",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-23T00:46:37.860401Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-22_17_46_36-7344037622406247747'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0823000710'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-23T00:46:37.860401Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-22_17_46_36-7344037622406247747]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-22_17_46_36-7344037622406247747
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-22_17_46_36-7344037622406247747?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-22_17_46_36-7344037622406247747 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:40.411Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.186Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.217Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.300Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.334Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.449Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.497Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.532Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.568Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.606Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.632Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.670Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.695Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.730Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.780Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.805Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.827Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.863Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.901Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.948Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:41.975Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.009Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.047Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.087Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.119Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.158Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.195Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.553Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.637Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.670Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.682Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.704Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.718Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.766Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.766Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.823Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.857Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:42.895Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:46:52.341Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:47:12.973Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:47:13.003Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:47:39.444Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:48:51.583Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:48:51.618Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:11.852Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:11.921Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:11.959Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:12.020Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:12.051Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:12.076Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:12.081Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:12.106Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:12.137Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:12.152Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:12.197Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:12.237Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:35.597Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:35.680Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:35.730Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:54:35.815Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:19.113Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:19.180Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:19.237Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:19.303Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:21.628Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:21.729Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:21.806Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:21.834Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:21.857Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:21.893Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:21.929Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:21.965Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:22.037Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:24.559Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:24.627Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:24.701Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:24.749Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:24.814Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:24.883Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:27.425Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:27.505Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:27.581Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:27.693Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:57:27.725Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:58:26.826Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:58:26.874Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-23T00:58:26.913Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-22_17_46_36-7344037622406247747 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598143594240/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.08655977249145508 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 18863eefbf0e42b4bdeffcc469c4533d and timestamp: 1598144416.6199296:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 822
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598143594240/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598143594240/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05321764945983887 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 822.980s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 46.107 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 15m 0s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/2ogdmi3fy7zve
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1833
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1833/display/redirect?page=changes>
Changes:
[noreply] [BEAM-7587] Spark portable streaming (#12157)
------------------------------------------
[...truncated 291.28 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWbbJvG8ACKN/////////////////////////////////////////////4AjT6gN26rgjZtCxsgE9g1E4g0UIZKYniNI9TTEeo9JtJ6mjTI0ZDQDTQNPUAAaAAAAADQAGgBoAAADQA0NNAGgAaDahkZDTQ0Gg02kGSEZBGRhNI0aJPSPUNHqabUAAADQNGmQAAAAAaNAAAAAAAANAAAAAAAAAAAAABkjYoAAAAEwCYAEYATIwAAmACYABBgTEYBMJkwAAAAAAAAEwJoyMCDAAACaEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRAEAgjE1T8hT9CCNB4oHqZqPUyabSMQAA0DTTQBkGmwpoAAAA0NAAAAAAAAAAAAAAABlUxtW6GMCzLlxYZfV6JIRPNCJo7VqrCjwDJUX2RQLF8uTJP168N7etfeZRAuyrQTeT0QuppiXPDEIDJf3xrBEICFCgQEjTEm0gGaegd9mkuYTGNd06J6pKbGNCUxCCdbyKAMWd4CwFAniAjQKg4L4QV7ljBnvrt23fxXl7czTTwFhpmAwmI8Z/g0kjw5nhE4Fqs0jUNmapxmTTDRLLi06UiwW8jOOgJl50p4YkFZhDD34qPEXBhtJlSKsZIO/07jMVBVOkmEwnBOQpWEphMCdZPx81OUBHQ+Ajh8To23Oc5KVXFipNzZxZGjXDMHQ0Ep0wi67hWzqymEJYYCMZKHJRIlIbCZpEpIbG8pBDYcaQo2xIZLQc0yG0yMZ18ZzBPCZxsgoNEmWLIGgaZSVkysps6wFgQznows5lUHXgkZkX/Tz+33/+rt8ibBMNrw7RT63wUh2uR20gJVgXA6k1+qIW+jso4YRz8H7oeLM30Sp1+6W0I3MR7JaYm52hpakMMMMMWCCBBBBBCCCBBEie8cEIhKAhgCiOACA9+n1BJSYoo/Sb/oNdctI4Jby4qBdNE01cwtWUnNkxN0MhZOHioLgYNOKi+WlRjbAWATc0cucs4t33CNZe/v1MxC9eLhtlQk8C4KjemoYVxJseHiWSzPAs5ZlI0E7x9gTgbOcg1MnuG4hQyQeaMoHQ140TCwuNGCYbMFSLjIihUFsURdOTEzzChRAU1WKBTMi7EcKDkeGuOLFNm0meOJEzlwwXNd5CSPUPSebZep0GHRWJYt1dlFG197Mv+WyZalf12PnOp6pUuBpbQmbMM0SVKgZxsboNZH7FPbVLdZDl5GU7ktL+2Z+89VXoGZgxW2TKsPbV8QfSfSZIklp0lFDVmpRLOjaAtBqyTtZRAJAo+vVaNset59oVRJr9wLFzuJJavSUrfsoZbF+GXd5CHurvxrixjKvGK49vHa82t1TgE0OYabVBJ1Q4CI41IQyWneqRA3y8lstSp3nEG0q/jl2ISIMGOFRqhcckHwYEk4gzhyIFDQ8WawllZewSgS85DqZZimoobiLpQKTXJSqDjqZlBXFErt/F+QVilUXRJ2k57meTVKd8VLrKqYVzHmU2gWXnrioovoBhdztne0djTFYklWqgwiymRXi64CtYmzj1LOdAGtBsiFBLoec0SF+iCJaRBkb0xlaaRtPgWwrr0vH0IVGYiAJQmJJEmB35b7bpTFgDc+CdUC0yDogQeSrsn1rL1arvGeORkObzEaZIruwY7VGAq3RyslJyKsoMjaMoMqdT1RAFBgGqESuCYBw4BRGdIuImqSUi7glRpHaCQw9wUh0h4Kw2NhUJqWOWuY65nQGwuwtiBMegLkBAoofJnBwzWIMocqGQyRQksbdm7w7x0HfC4gx8lYqxY1yC3ytVDd1K6spVKKL1hQKmmyHXUKFVN0SbqqNFJLAv+2W2EzGRvydu+qKp4ZciEVhika5TpT5sVNwtSnMyl0ErYiZF2e0DpmgjmVBMDZMFBo0k7FjcXip5NUWlw9QKqaX4vbm0AvrlF+Z6AtekqBvbFtCsXAWqAlBwJitgmAeCwttdKbyYIMA1eRSk6+Who8rMLRCgw/l6PdKTkMQdBCiACgF6DBrCL7ndlRMwUo4mpuIkpx1IAmjSir6gk/wtvhKo2WWomIKbf4Fc7D5mQs4Did98+3DYBuAYFpAnpmcc2oB9UC+dMPDy9RVie6i2JsRRQaGA+AcpdyZwLAUjlGrmcqMrt3qs66c0NDbNXNLKEZDLYKrNdPmxwMuq0jvFrKERIaphHHX1Kgg7OWPFJhUqG0IxCBcxAbGsDHswGLBULplOQgINVwoYEXgsyhXPjJSvIvswTfpeUktYNGgNYzrtKynGECrYoueAeG3fp5+KsqrhqsSbBLM1NSWcrQZNTBlKbdxToQTFjwkWHLFapa4qot4F4r9faUBWNGJoprC8KFcWNTQnSQRRAGFyFP2eCYm27mWwUsJKlq0UQ5lqYyTUEK3emOc9A1oN1zWgKWHue3YtmkEbI2DR0SkzVquadgYCShp2FEKe5zZZaDs41MxFGZUpFtW1jLjVhuLKDYEUtXIzM2q00CxnRqWCGwbNeApV+0itKuMEbJSryMBszyxNtNVLnPV9aplnYyGpc/D5Xn4iKHO1Fa15ku31jmQuc6sx0Dz0ttEgmSSCCCAwV2sXQp5bssjER+WWRS0t+Uxlf3LLrkL1h95jYl3jynoYbaYSRQqEkagymp/UHoyoeHiYQBgwlKxQAAAAAAAAAACwABoABGLFkwoaBAAgggggjeC2JPPyWGfn221RFcm227bYk7bbS22666nNIvvyypviLnLLlxSbbbboovlVKiTbbduSUuErZziIvyylTNFjFmwSNniKNFpSmlttvrTwJrTExjNVVmua0BR9nVv4x2bvDhBSErC7cLc5sU7Lah1wlyDBRWONurBU34yDmw0DRNNVC3JS90ObGpbtdSwEli0tjdn5m+yQfgC6ZmisC4YtIOhHHZANQYpkmMz1kTClVpcgtJetmLwLhmgWDBmkGV7G38IUGoM9IM7KG6yGMvlvGlMJyzbGGjK0WXAppD8IIWZ0UCHG0rPKZbjBS64GgqrmEtDhYs2WoEwzHWlE0czVBBlYr55r/sQt1oIifgxSxTDjbs5M9Z+OgrAQOlODUKIhEhFvA9rMZiz8v1/zqdJGHZBM68SbFoL1IF3GPWo11p8DSSNJXUkjCjZ8epI06tUlKtbWw1SkySW2Hc4YNq0osYSqKMJuFRTTXaVK/dqdhvWhsmvpMU1gI89UQ6CmDECRBhIQQCyCBn6HDjlu/rrjZ+9MV6fb7q3M+/4fGwzfyXTi75/T6zqv7K2P4/M9+v3/Z//UDmhgbuIAfisICLAbgAhAhGEEEFADAMJBIKwDMC04Cjif/1sRAcoRFBEuUaF1FBQObWtBngg8ChTeAlEH3o/YMkDYNMGwaYNgxfciA2QOkggUhoBtJDbaB+5gRzrQeVaFWNLW9T/8XckU4UJC2ybxvA=",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-22T18:46:33.475160Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-22_11_46_32-4942580203123012995'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0822150253'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-22T18:46:33.475160Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-22_11_46_32-4942580203123012995]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-22_11_46_32-4942580203123012995
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-22_11_46_32-4942580203123012995?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-22_11_46_32-4942580203123012995 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:36.102Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:36.933Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:36.971Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.036Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.065Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.167Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.209Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.236Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.284Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.307Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.342Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.375Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.409Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.443Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.480Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.516Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.538Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.571Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.594Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.616Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.643Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.671Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.702Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.740Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.777Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.810Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:37.832Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.184Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.255Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.287Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.300Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.315Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.338Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.402Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.402Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.464Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.491Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:38.523Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:46:50.106Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:47:05.226Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:47:05.264Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:47:36.614Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:48:45.113Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:48:45.150Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:02.792Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:02.866Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:02.889Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:02.963Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:03Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:03.024Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:03.027Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:03.054Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:03.082Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:03.097Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:03.123Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:52:03.159Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:54:17.863Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:54:17.925Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:54:17.964Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:54:18.007Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:55.748Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:55.818Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:55.862Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:55.921Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:58.083Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:58.154Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:58.210Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:58.245Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:58.264Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:58.297Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:58.317Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:58.366Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:56:58.442Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:01.564Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:01.638Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:01.709Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:01.767Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:01.941Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:02.001Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:04.326Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:04.398Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:04.479Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:04.705Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:04.736Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:57.301Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:57.348Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T18:57:57.388Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-22_11_46_32-4942580203123012995 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598121989730/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06302595138549805 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 66c2458fffd446f9afe55683062a4fbd and timestamp: 1598122794.9020045:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 804
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598121989730/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598121989730/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06672096252441406 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 805.773s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 28.377 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 36s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/rqx4z3zdm32gi
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1832
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1832/display/redirect>
Changes:
------------------------------------------
[...truncated 290.30 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWdDExRoACKN/////////////////////////////////////////////4AjT6gN26rdTHNtCxsgE9g0k4g0lGgSYnpo0m0TENPSbSep6T1PRNM0jIaNPUDT0QBkBoBoAAAAAGgBoAADQMQ0NDQAyABoyeoYCNPSNGmQzJBkhGgTEYjSYmhT0j1DR6mm1AANAD1A0aZAAAAABpk0AAAAAAAAAAAAAAAaAAAaAABSVP2Sj1BDATJoHqGJgEaZMEyaDCMJkyYAEGEwDQmRoxGmhk9CMRkw0NEGmTEaBiDAmE0YmQMmmmBGmhoaGAQep/qkGITACNMTTABMEaZMBGmmTAEwBGACYAIwARppiYATAAAjTAAJgAAmmAAIMIaNAwAJMABMAATAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASJEARMIIxNU/I1T9CJ6jQfooHqZqPUyabSMQAA0DTTQBkGmyamgGgAANDQAAAAAAAAAAAAAAAZRMZTu5iArS5cVWXI6JIBPNAJorFinBivjJSXWQvrFxcmRdjjgtWmuuMofXWVopvJ6IHq9eFc8EIfMl7cNQIhAQoUCAkaYk2kAzSzzv00lyyYxruvQPUJTYxoSmIQTr2KgDFnewsBQJ4kI0CkPBfCiO3Vv57l27ZvYby1bzjSwFhpGAwmYdH/g0UjxBnBE4FqM0TTNqahxmPSDQLLi06UiwW9jNOeJl50p4ckFZhDD38qPE3BhtJlSKsRIPAU7nKVBVOkmEwnBOQpWEphMCdZPx81OUBHP+BjidD0TbnOclKrixUm5s4sjQrhmDn6CU6YRddw7Z1ZDCEsMBGIlDkokSkNhM0SUkNjeQghsONIUbgkMlnuaZDaZGI7AM5YnhM02gUGgTLFjDPNIpKyZWU2dZCwIZzkYWcuqDr4SMqL/p53cL//W7ERFwn1eFb2Dyxj5OZ6PjOAm0xSXdo6quFypDPsk0g9h77Hq0uJE2aHjb2D8YHiy/VmzZzr9oooooo6GMQhCEMYxCJJzj8KuigVgELYBAMwkKNd2zAt9JwOe1lykjhFvKioF00TTVzC1ZCc2TE3QyFj4mZQXgX9RKi6WlRia+WATc2cucq8W5iw3BMbmYeazDLiqS8oscLhKN8T0jhqdr3cuhlld6zVkUjPTvH2FOBs5qDTyW57oFDIhxoyYegjiRMHB44X5grP2EXERDAoLYoi4ckJnGFCiApqsUCmY12M4cHH8RbwsU2a8zxxImcqGC5rvQBE1xBr2zfxPgQBJ0Iyf6MstQ4dlf/xOOwyiBYonuI2UqPSgIEbCYMAMEhGgBjaTSazcSO9R8Kz8bNy2cmvJ8J6gov1U2l2BjGBhDKFEI7qng88k8koPkd0RFlkFEFnrKSzgPAEFBO2oK6AOiDL7faRjH1KvejYCEA4BhqaXyyxrV0SOstkT6Z5xk18a4ykanwcFHIZFjIbsyrAo31Ge6hkao8c3HpI+JR1PZ3Zuzqsw+GLfbVJ5mS5TBeueTg4gmT5E/AFKkeeBjSRwpjLyIFBr4kyIkVrdwSQHech1MszJqKG4i6UCk1yEqg6dTMoK4oldwIvxisUqi6JO0nPdTx6hTvypdaVTCuY8qm0Cyc5cVFF9AMLubs74jsiYrEkq1UGEWQxq8XXQVrE2bxSzXQBrQawgQS6PntEhdofhWkIZE5MZWmuNs8KyFdgl46hCojCQC+aCwvAelmfx6h/egB0/p5p8QIREoiuch4nkYvj6D1WBvq+07zuz71LpfCYTNeAtXZ0Ex25dCS4jKVC4rJZUBAEuAMgEQlCMAy8AojNkXETVJKRdwio0TtJIYe4KQ6Q8HYbGwqE1LFLWMUZnQGquqtiBMcgLyAgUUOkzg4RqEGUPKgkMkMBLE3Zs4d66DvpcQYuQsVYsS45b9Wqhu6ldXUqlFF6woFTTZDrqFCqm6JN1VGgklgX/bLbCZiI4BO3f1FU8MuPCKwzJGsU6M+ZFTcLTpyspdBK2ImRdnNA6ZoI5dQTA2jBQaFJOxYnF4qeRVFpcPTCqml9Hrm2AuW6L0z0Ba8goG1VsoVi2C1QEoN9MVsEwDgWFtjSx9JaEwVOYsCOHLXOk3PtggQYPhTlDNM4Mg2iBD6EgEcEyp/cIzBICsQKaZUdLCzDNuIAlKmuuFmOfIG+v8ArRRZJwVK+LEreKYxC7gMqekq2IVgMQDARQFTRM2sWA8pA4U5g5ObswuTu/KyVkTAKTAPAHJXcicGwFI5Jq5nJjK7d8rOvHMDQ2zUyyyBGMyWCpznUpsaDLrFcd8tZQiFDTMIoo9VoIO1FjhSYUqgsCIQAW8MGrrYxzL5iwVC6ZTj3yDTbKF9F37MmVz4EpXiusvzepeUktUNCgNUzbtGynEECrYoueAeG3gJ52ZWVVw1WJNgllampLNVoMmpgylNu4pz4JixYSLDlCtUtcVUW8G8V+ttqArGjMaKawvChXFjU0J0kEUQBhchT9ngmJtu5lsFLCSpatFEOZamMk1BCt3xinPPNWDd8xnilh7prsW1SCNobBoaBSZa1XNOwMBJQ07CiFPdZcks92camZhRlVKRbVt4yYlYbmyg2BFLVyMrNutJAsR0SlghsGzWgKVftorSrjBG0Uq8bAbM4sTbTVS5r1nW6ZZuIhqXOw+T52Iih57VFrXmy7PXPMgeeeqMefcclsoj8ySPvvvl+NrF0KcW9lUYiOyyyKWlnzGMjzNZ63A5VdcY2Fd44pyCCymCkMChJEoMr0u6m5EVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEEbwWxJ52Owzs622qIrk223bbEnbbaW23XXU5ZF9+SVN8Rc5ZMmZJtttuii+VUqJNtt245S4atnOIi/JKVM0WMWXBI2uYUaDSlNLca+rPAmtITGM1FWaxqwFH2dS/jHZs4cIKQlYXblbrLmTstqHXCXHMFFY411YKm/EQcyGeaBpKoW6KXuxzY1LZXU8BJZmjsbJ+ZvssH4AumZoqAvDErg6Mcbjw1BiGSYzOVhMK6rC5BYS02YuAvDLAsGDLIMj2NfwpQaYz0gzswbvGYi+W9aUwnLLsYaMlRkrE+BTaIks1wIrWhGzn4tOZn16ZUKMKtVRWY8mRqBMIxVJRNFM1QQZSG44132oG66ERM0xSxXgxl2cmeo7FQVgIHpTf0iiEQoRZvvc7Lsa3zfZ/OrVyMKsCZ14c2JQXK4F3ActRrLS4OikaKupJGFG13ikjSq1CUq1t7DUKTHJbgdzhg2rSixhKoowm5VFNNdpUr9lTsN80Nk19JimiBG1T8G0PQyYlQYHBaAiWgxpxkylnH5W1W9zrCdoKGGoqT6fW40v2Ss0tN9/xTofyhT1H7qar+f6rrGjWtWYQPoqIQ8IkBiACCwH38LQkAMAYHA4IQDGAisY6CrrL0fAanoVpENT9a3Bo0a3lsBjR4chWo3IJRB96P2DJA2DTBsGmDYMX3IgNoDpIIFIaAbSQ22gfuYEc20HlWhVjS1fU//F3JFOFCQ0MTFGgA==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-22T12:46:35.351530Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-22_05_46_34-74424118897812593'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0822100253'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-22T12:46:35.351530Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-22_05_46_34-74424118897812593]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-22_05_46_34-74424118897812593
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-22_05_46_34-74424118897812593?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-22_05_46_34-74424118897812593 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:37.761Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.472Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.505Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.589Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.618Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.733Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.779Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.811Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.842Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.863Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.896Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.935Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:38.987Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.009Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.043Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.079Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.108Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.134Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.162Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.197Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.230Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.269Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.306Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.345Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.378Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.422Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.444Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.749Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.820Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.846Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.870Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.881Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.906Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.966Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:39.966Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:40.042Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:40.069Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:46:40.145Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:47:07.097Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:47:17.075Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:48:52.650Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:48:52.684Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:09.808Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:09.889Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:09.924Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:09.998Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:10.027Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:10.060Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:10.061Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:10.080Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:10.119Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:10.119Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:10.158Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:52:10.196Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:54:32.539Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:54:32.614Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:54:32.669Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:54:32.734Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:14.530Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:14.599Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:14.654Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:14.723Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:17.225Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:17.306Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:17.378Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:17.418Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:17.437Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:17.502Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:17.507Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:17.587Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:17.656Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:20.174Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:20.244Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:20.316Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:20.371Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:20.454Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:20.531Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:22.916Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:22.989Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:23.103Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:23.160Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:57:23.200Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:58:21.415Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:58:21.469Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T12:58:21.503Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-22_05_46_34-74424118897812593 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598100391690/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.059888601303100586 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 0caf62bc1ad8410da13e773ed3003d3f and timestamp: 1598101206.8851027:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 814
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598100391690/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598100391690/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05418276786804199 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 815.700s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':' Thread 4,5,main]) completed. Took 13 mins 38.704 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 51s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/mc73i6baya6y4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1831
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1831/display/redirect?page=changes>
Changes:
[noreply] [BEAM-9918] Adding tests and documentation to xlang components (#12667)
------------------------------------------
[...truncated 290.57 KB...]
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWfVjb50ACKN/////////////////////////////////////////////4AjT6gN26q4Y2bQsbIKT2DUTiDRQhkpkemjSPU0xGj0m0nqZNMjRkMgNNA09Q0Bpo0AAAAAGgAaAaAANAAaBoaDQBoAGgyGmINNDQZDJ6QZIJoEZMmQjRo1Go9RkaabUAAADQGjTQAAADQ0GjQAAAAAAADQAAAAAANAAAAAABkjYoAAAAEwCYAEYATIwAAmACYABBgTEYBMJkwAAAAAAAAEwJoyMCDAAACaEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRNAhMIIyNU/KNT9CR6jQemoHqPKPUyabU09EaAA0A00AZBo2KaAaAAADQAAANAAAANAAAAAAAGUTCs5UxgTS5gVmX1WiSEV54RPHYsWcXHgGSyXmRQLF8uTWnqtWG9vXPNZRAuZWhm8r0Qv00xLrwxCAyX98agRCAhQoEBI0xJtIBmponfppLhJjGu6eFeqSmxjQlMQgnXsigDFnLBYCgTxIRwFkPhfCirbrYNe+uXLV/FeXtvN9TCWmoYTEZx0P+DTSPEmgETgWqzTNobM1TjsmoGkW3lx0RFouQjPOdJmA6I8QSCwxBi7+VnirwxXEytFeMkHgKbfMVhXOhMJhOCchStJTCYE7CfkJqcoCOe8DHD4nQNuc5yUq+LFabmziyNKyGYeeqJTpCL7+funXlMQSxQEYyUOSiRKQ2EzTJSQ2N5SCGw6YQo40kMlouaZDaZGM5UZwieIzzZBUaRMtWQNE1ChYTLClvWAtCGc3GJnLKo6+EjMjB0mh2/B/6xX6GsvuaXqlw7u1wnH7fJ7iQEsvLg9X30NKLfx2WmF8c8h56Hjjb+JY7Jd7iEYoR7JaYm52hpakMMMMMWCCBBBBBCCCBBEie8cEIhKAhgCiOACA9+n1BJSYoo/oN/zuuuUkcEu7AKoXSxNNXsLllJzZMTdTIWTh51RfBg1AqLxcVGN0BYBW5s5g5d8uxR4CotuYXjixq7BRSWnprgqN4bQ59V0q4lmMlxHnvCtBZlI0k8Q+EnA2d2g2uT3TdQoZINcMoH4asaJhYXHC6YZoLxFxkRQqC2KIuHJidrCpRAUrtUCmZFzBw4N9xFui1TZrTPHkiZyoYb2u9hi/FTkPZ6D3oxlVuZlqlWCDOtt8r4f0xj55at5FGaTE5dsoRnRG7dF1ZaqBnGxujj9fs09xWJDJ8zKSvdl5manPxQVt0BmYMQrFjXnVm7IO5O5MUSS1SSihqzUpVsRahaDVknElCAkC2FjrkbaNfJ21VDG4boWfncXpR8dUeAxglsR4plx48HFuMjdmEZXYxXsGun2Rs4tN7TQkjTZoJN6HBREjT0MrqnSnvl/vZbXWKd+Yf2Vd8kywCRBewxCNaLkEg7DBknCGcGRAoKDiDWEsr7qCUCXnodbLc6aipuIvlApNb6VYdNpMqLIqlfv4wZBWqVZfEncTnuJ5NUpvStdZVbCyY8ym0Cy83eVlWCoGF/NW98RzCYrUkrFWGIWUyLALroK5ibN0pZ7qA5wMwhQS6LndEheogiWkQZG2cyuNI2zwbQVmUvHkIVGYiAJQmBJEmB4Jf77xTGgCQ8M+oFpkG5Ag5lYpTr2rrVvfPGQRkebJI1PnruyY7dNArnRy/SlJNWT+RsmL+VVqyogCfwDVCGXBMA4MAojPkXkTVCUi/glZpnaCQw6MoHoTwdpsbCqTUsctcx1TXQGsustiBMegL6AgUUPE1wcM1CDKH1QyGSKEljbt3eLkHUd9LyDHvrVYLGuPW9VyqbvourKVairAsSBUpbDsrFCrm6pN11mkklhX/bbrSZjI35O7e1lc8UutBFgZ0jXKac+ZFS8W0pmZR1EroiZF+g0DpNBHLKCYGyYKDSoTtWNxgFTkVVcXj2gV0o+h2xtQL63RfmvQFr0tQN7WtIVi2C5QEoOBOVsE4DQsLdVSqct+C8M3MVFOyFo6dLe5aIEGHovTNys9BEG4QIewoBfAvZwDhFXhUTwClZj3zsJ055vIAmTKmOFXSeoG5wFaaLLeTAFOvT4upY+hkLYBMT/womoaANQDAtwFBOTza3gO6gcJ8w5ObpD2d1fWhNCKHDIwHYDkr+ROBaCkck1ezkxll28VvXjlxobZq5pZQjIZbRWc41GfGgy6vSO+WsoREhsziOOrqlBB2osaUnFlUNgRiEC3jga2SGPZgMWCoXDKchAQbNsoYEXgs10rr3ZSvIvMwTfpgKEtYNKoNYz79O2mMIFYxRe8I8V2/T0M6wrshqwSbBLM1NSWergZNTBlE27ymjBMWPERacoWKjXFVV3AwCwa+1qCwaM5opYGAKleWtTQnQgiqAMTkKftsMxNt3sugowkqNXCiHMuTGSaghXbwxznomtBueX0RSxdz2zFs0gjZGwaWkUM1hfYUgAqBZUhSANUWYc8y6UgxhQSBs5eJELrqppTAbe2o2BFGr0ZmcYtRAsZ0ClhhsGzXgKLBtYsSsjDGyUrMjAbNAtTbTVa7r67rdJZ+MhqXOQ+T5yIip9/U1rXil2uufZC++/UY/A1stpEgnSSCCCAwVXMXQpq38ujER6WWRS0teYxlX3Jn7cLazzWOiXeNU2GG0mLSKFQkjUGU2XtSbGVDw8TCAMGEpWKAAAAAAAAAAAWAANAAIxYsmFDQIAEEEEEEbw3RJ6GS00NC66uIsk223ddEnddcXXX330zSMGDLKmCIvcsuXOk2223VVglXKqTbbd2SUufV05xEYMspUmi1izYZGzzirSaUppcbttaeFNagmMZqqw1zWgKvs6uDjnbu8WIFIStL9utxmzp23VjshLj2CiwcbZWipgxkHMhomkairFuCj3I5salu11LCSWdp7G7PzN9kg/AFwzuFQF8YlIOiHG5AOQYhknM7ZhOKVWFyCwl66ctBfGaBYcOaQZXsbbwxUbQZ6YZ2UNzkMZglyDSmE5ZtjFVlcJnwU0p6EELNdFAh82FZ9TLbYKX7YcCqqYS4PitNM5AmGY6konjncoIMpFfNc9moXZMIifgxSxTDhXK5Neo9HQVgIH5Tg2RREIkItYHt9jmZvl+v+dVpIw5gTXXjp8SgtpAuXW3I11qcDTSNNX0JGJGz3SkjUr1SUrFxlpqlDJJcaO9wwbVxVawlWVYjbqqlLLitYN2p2m8aGya6RimsBHoH0NwqYxQkwYSD8AsfgZ+r1+uScfZYWjzLw8vNTcLOfD4/KzTvzXTy76fX7T6v7q2H5/VBQ/v+0VH/m1I+Rgex+q8gIgBqACD4EYAPwUAMAwkEgrAMwLdgqcKKjiUQGyEJ/oZsjSN4bNqS4LQZ4QOQ9ibkEog+9H7BkgbBpg2DTBsGL7kQGyB0IIFIaAbSQ22gfRwI5poPLNCsGlrer/+LuSKcKEh6sbfOg==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-22T06:46:27.898913Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-21_23_46_26-17795461546563677192'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0822050257'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-22T06:46:27.898913Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-21_23_46_26-17795461546563677192]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-21_23_46_26-17795461546563677192
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-21_23_46_26-17795461546563677192?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-21_23_46_26-17795461546563677192 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:30.234Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.048Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.087Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.168Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.215Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.313Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.361Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.398Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.431Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.466Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.504Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.529Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.560Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.599Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.641Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.667Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.706Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.733Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.768Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.795Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.835Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.863Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.893Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.925Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.956Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:31.987Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.017Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.498Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.569Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.595Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.606Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.627Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.640Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.699Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.699Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.759Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.795Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:32.832Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:46:54.366Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:47:14.075Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 7 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:47:14.107Z: JOB_MESSAGE_DETAILED: Resized **** pool to 7, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:47:19.419Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:47:19.449Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:47:24.750Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:48:38.175Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:48:38.202Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:39.936Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:39.975Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:39.990Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:40.025Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:40.041Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:40.057Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:40.061Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:40.083Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:40.092Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:40.093Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:40.117Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:53:40.133Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:54:14.297Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:54:14.356Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:54:14.391Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:54:14.440Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:15.617Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:15.683Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:15.735Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:15.811Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:18.212Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:18.267Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:18.329Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:18.356Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:18.376Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:18.410Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:18.432Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:18.480Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:18.531Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:21.036Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:21.088Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:21.134Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:21.185Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:21.235Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:21.276Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:23.874Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:23.960Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:24.044Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:24.161Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:57:24.184Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:58:17.935Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:58:17.973Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T06:58:17.994Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-21_23_46_26-17795461546563677192 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598078784182/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.0816342830657959 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 34b703c79b86455894c4aa3cad3b7afb and timestamp: 1598079595.504683:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 811
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598078784182/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598078784182/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05208134651184082 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 811.881s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 34.608 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 38s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/xs6a5mrg2cpuw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1830
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1830/display/redirect?page=changes>
Changes:
[Robin Qiu] Fix bug in JoinScanWithRefConverter
[ningk] [BEAM-10771] Added Whitespacelint job to pull request template
[noreply] Use setCoder, not setSchema (#12662)
[noreply] [BEAM-10703] Prepare Dataflow Java runner for shardable states (#12578)
[noreply] [BEAM-10549] Improve runtime type checking performance for the Python
[Kyle Weaver] [BEAM-10460] Remove SparkPortableExecutionTest altogether.
[noreply] [BEAM-10788] Also disable immutability checking for transforms within
[noreply] [BEAM-5715, BEAM-8862, BEAM-8702] Removes all references to grpc_all and
[noreply] [BEAM-10761] Prefer TestPubsub#assertSubscriptionEventuallyExists when
[noreply] [BEAM-10654] Implemented ExternalSchemaIOTransformRegistrar for jdbc
[noreply] Merge pull request #12580 from [BEAM-2855] nexmark python suite
------------------------------------------
[...truncated 291.59 KB...]
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWeFWTx8ACKN/////////////////////////////////////////////4AjT6gN26rdTHZtCxsgE9g0k4g0kIaSYnpo0j0TENPSbSep6TT0EbUMho00DT1NAGmmgaAAAAAABoAAANA0DQMjQ0AGgAaNNqGTJo00PSaaDMkGSEATEYU8gmmpPKPUNHqabUAAADINGmQAAAAAGQAAAA0AAANAAAAAAAAAAAAABSVP2Sj1BDATJoHqGJgEaZMEyaDCMJkyYAEGEwDQmRoxGmhk9CMRkw0NEGmTEaBiDAmE0YmQMmmmBGmhoaGAQep/qkGITACNMTTABMEaZMBGmmTAEwBGACYAIwARppiYATAAAjTAAJgAAmmAAIMIaNAwAJMABMAATAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAASJE0CEwgjE1T2hT9CCNB4oHqZqPUyabSMQAA0DTTQBkGmaamgAAADQ0AAAAAAAAAAAAAAAGVTCp3UwgWJMuK7LkU8b4mlfEsN9fU4MOAY6S2yB5YuLjyTcUT9q0420yd5dhWhmzNO+7RRAuZ+APGO/uGsEQgIUKBASNMSbSAZqaJ32aS5ZMY13ToHrEpsY0JTEIJ1zIIAxZ3kLAUCeJCOAUh0L4URW696a5du1X8Fm1bzbUwlpqGExGceM/waaR4c0AicC1maZqmzNY4rJqBpFt5cdKRaLexnnPEzAdKeGJBYYgxd+KzxF4YriZWivGSDv9NzmKwrnQmEwnBOQpWkphMCdhPx81OUBHP+Ajh9F0bbnOclKviRWm5s4kjSshmHn6iU6Qi+/obp15TEEsUBGMlDkokSkNhM0yUkNjeUghsOLIUbckMlouaZDaZGM6+M5YniM82QVGkTLVkDRNQoWEywpb1gLQhnORiZy6qOvBIzIwfT0O34P/WiQibVPLr+4xT0yxEp3+d4EAJpSUn2fl+6oXFkc0zUyL4HzqefQYMTRoi+XYHxoeLL9WbNnOv2iiiiijoYxCEIQxjEIknOPwq6KBWAQtgEAzCQo13bMC30nA57XXKSOEXcqKoXTRNNXsLllJzZMTdTIWTh51RdBe1AqLZcKjC48WAb7mDlzv7pcn4DCMTGNrxCyp5kFmoyIqiFwlG+NUrXDn0Th5ngWesykaKdw+wJwNnNQauT3HcgoZINODKB1+KFEwcHjheL9h6hFwkQPqC2KIuHJiVphUogKV2qBTMi7EdDBx/DXGFqmzYmeOJEzlQw3td5AImuINe2b+J8CAJOhGT/Rllp/Dsr/+Jx2GUQLFE9xCpKj0oCBGwmDADBIRpAY2fz+n2e13KO+sEllpXLzPhnJ340H5poN3BjGBhDq1dkeFryc9E9ErPJDoiLLIKILPWUlnAeAIKCd74K6AOiDL7faRjH1KtmGwEIBwDDU0vlljWrokdZbovzTrnKMY5z5kaqj7HHpdqykduZbHIvyJ0sZFacg4HSZ45jsdmNc7uy+QwBbrYJ/GzYLLH6J1UECClViz4IuaRzyMOSMMYzEiBQrGLMmJNZXgEnCHnIdbLc6aipuIvlApNchKsOMpMqLIqlfwIwZBWqVZfEncTnup5NYpvytdZVbCyY8ym0Cy85eVlWCoGF/N297R2NMVqSVirDELKZFgF1wFcxNnGqWe6gOOA2A+gk0XO54y3O9AtIAyFqUyOGgbV4FQV16WW0IVCYCAJOSokCRA8k5SSSfIgDg+VonDcYg3nEHUrR1+3b+1B8Ot5Q+U6Mse9y6jumHfqwFi6ubmOvLpSYEZYrYFXa8kIAmABkgiVARAMxAKIz5F5E1QlIv4RWaZ2gkMPcFA6Q8FabGwqk1LHLXMMRmQGuuutiBMcgLqAgTztkzA4ZrEGQOqfjMcD5LG3bvMW9dR3wvIMfIWqwWNcct+rlU3fRdWUq1FWBYkCpS2HZWKFXN1SbrrNJJLCv+23WkzGRwCd2/rK54pceEWBnSNcppz5kVLxatMzKOoldETIv0GgdJoI5dQTA2TBQaVCdqxuMAqciqri8eqFdKPxe4NoBctz35mnC16WoG1XqQrFsFxQEgOBKVsEoDQWFuRJXOfABSFrqKyjEFqKvNTzcIQGD6U3PzNHCkG8QgfgnAwIUrYFyjb6kKtgpln8qKGmKNwIAliyquVfQfYS8QMEuTRhKgVE+r1DiPsYhcQGbShp2QXAMgDAXMCmoKNtGAPScOTQwdHV3irY8P64lxFFBYYB5A5K/kTg2gpHJNXs5MZZdvlb105gaG2a2aWUIyGW0VOb6jLjQZNWoHeLWUIgQ0yiGGLVJyDtBY0UlFKn74Qh8C3iA19ZGOZgMWCoXTIce8QabZQvIu8zKFc2MkK8g2y8b9MBQltQ0qg2pn36dtMYQKxii94R4ruAnoZ1hXZDVgk2CWZqaks9XAyamDKJt3lNGCYseIi05QsVGuIqruDgFg19pUFg0ZzRSwMAVK8tamhOhBFUAYnIU/Z4ZibbvZdBRhJUauFEOZcmMk1BCu3xjnPRNrBu+Y0RSxdz3DFs0gjZGwaWkUM1ismnaGEkoadpVCnus2WWi7eLTM4qzKiRdXtoy41abm2o2BFGr0ZmbZaiBYzo1LDDYNmvAUWDaRYlZGGNkpWZGA2aBam2mq1zXq+tUln4yGpc7D5PnYiKnXdTWteaLq6x1j7rrtZjrzTUlSI9KkbzzzxeicYudTS3d+RiI3JJGpaVeUxkXuWHbb7VdtpjkC7LSmn36kwUgfUI4VBlFLepNQlQ8PEwgDBhKVigAAAAAAAAAAFgADQACMWLJhQ0CABBBBBBG8N0SehktNDQuuriLJNtt3XRJ3XXF11999M0jBgyypgiL3LLlzpNttt1VYJVyqk223dklLoVdOcRGDLKVJotYs2GRs84q0mlKaW33G1nhTWoJjGaysNc2sBV9nWwcU7d5ixApCVpfuVus2dO26sdkJccwUWDjcK0VMGMg5kNE0jUVYt0Ue7HNjUt4upYSSztPY3h+ZvskH4AumVwVgXRi0A6Icbjw4gxTHKZWrAlFCr5cYvktOSloF0ZoFhw5pBlexuPCFRqjPSDOyhu8hjMEt60phOWbYxVZazLYJ8ClwiSz3AixaUbzRx6szRs1SsUYlcqywyZcriBMMw1pBLDK4oIMrBcacb9h9zWwiJ+DFLFD+FdmJmrNwzlYCB2Q3qRPAIEIqwPazOZsfL9f85GgjDsAmZeJLizlqgC7jGrka61ODppGmr6EjEjZ8apI1K9YlKxba01ihkktuO9wwbVxVawlWVYjcqqlLLitYN4p2m+aGya+kxTTAjaV/DeHsY4S4MEBAAJoAGNNs2cq5++zrnz2xXt+M/D0FD8/pa6L6qKNR9vv+GiWkSqqX9U37/n+p6hq2qV9+D8KWyHCLAZAAgvB8CIAJwDAGCAQCUAxgLnhquKeoup4DY6GaxLY+pcA1atro3AxsQHQVKLoEog+9H7BkgbBpg2DTBsGL7kQGyB0IIFIaAbSQ22gfuYEc20HlWhWDS2vqf/i7kinChIcKsnj4",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-22T00:46:56.656851Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-21_17_46_54-2519339669040960941'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0822000637'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-22T00:46:56.656851Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-21_17_46_54-2519339669040960941]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-21_17_46_54-2519339669040960941
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-21_17_46_54-2519339669040960941?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-21_17_46_54-2519339669040960941 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:46:59.276Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.014Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.053Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.120Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.152Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.249Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.293Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.324Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.356Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.385Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.418Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.451Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.477Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.511Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.530Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.555Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.589Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.625Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.664Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.692Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.735Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.766Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.798Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.850Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.883Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.922Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:00.958Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.183Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.252Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.286Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.293Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.327Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.337Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.387Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.387Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.484Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.519Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:01.720Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:17.692Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:28.230Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 7 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:28.409Z: JOB_MESSAGE_DETAILED: Resized **** pool to 7, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:33.638Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:33.679Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:47:38.940Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:49:04.256Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:49:04.292Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.159Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.227Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.272Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.353Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.390Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.409Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.426Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.445Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.465Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.480Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.510Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:52:24.545Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:54:42.508Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:54:42.594Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:54:42.646Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:54:42.721Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:22.526Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:22.607Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:22.667Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:22.757Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:25.158Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:25.270Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:25.350Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:25.393Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:25.412Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:25.442Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:25.506Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:25.541Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:25.626Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:28.113Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:28.214Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:28.289Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:28.333Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:28.402Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:28.496Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:31.008Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:31.102Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:31.237Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:31.313Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:57:31.348Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:58:23.623Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:58:23.677Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-22T00:58:23.709Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-21_17_46_54-2519339669040960941 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598057212033/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06270146369934082 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: 0ede2197deb24b1f81f7c6c306532576 and timestamp: 1598058022.8185709:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 810
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598057212033/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598057212033/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.11221504211425781 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 811.428s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 35.933 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 15m 4s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/6lfmkdmkas6fe
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1829
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1829/display/redirect?page=changes>
Changes:
[Udi Meiri] Color Jenkins logs support
[ningk] Added a whitespace lint precommit job.
[ningk] Removed trailing whitespaces from all markdown and gradle build files.
[ningk] Updated jenkins precommit job README, added comments of what needs to be
[ningk] Removed trailing whitespaces from HEAD changes
[Alan Myrvold] [BEAM-10049] Add licenses for go dependencies in python, java, and go
[noreply] Propagate BigQuery streaming insert throttled time to Dataflow worker in
[tysonjh] [BEAM-10751] Revert "Extending archiveJunit post-commit task with
[noreply] [BEAM-10764] Make is_in_ipython robust (#12641)
[noreply] [BEAM-10766] Fix flake where ByteString coercion could modify cached
------------------------------------------
[...truncated 289.72 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWVbx+hUACKN/////////////////////////////////////////////4AjT6gN24udMdm0LGyAdvXCkniDRQhlJiemjSNNMR6T0m0noj1MjRkMgNNA09IAaAAAAAAAAGgAAAAAyDQ0AAaABoyeoaaaDTQ0GQxpBkhA0TEYQmJok8o9Q0epptQAAAMmjRpkAAAAAaG1AAAAAAAANAAAAAAAAAAA0AAGNTxJ5SNBDI9EaYg0NNDTTINpNMmj1HpBiGmTCZMhpgRpgRiBpoxDCZNHqaGJtINoQNBppoBpoyAaaGBGCAaNDRkyaEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRNAhMJMjE1T8hT9CCNB+igepmo9TJpiMQAA0A00AZBpmp6mgAAADQ0AAAAAAAAAAAAAAAGWTBsbsYgK0uKKrLkdEjRPM0TRV69jAivjJYLrIYFi4uTJuxxttWnnXGUQLrK0U3k9DX6aYVzthEBkvbhqBEICFCgQEjTEm0gGaegd8mkuXTGNdy6B6pKbGNCUxCCddyCAMWd6CwFAniQjwFgPhfCiO3Vv57l27ZvYby1bzjTwlppmExGceM/waSR4c2gROBarNI1DbGqcVk0w0S28uOkItFvozzniZgOkPDEgsMQYu+lZ4i8MVxMrRXjJB36m7zFYVzoTCYTgnIUrSUwmBOwn4+anKAjn+/xw/B9E25znJSr4kVpubOJI0bIZh5+olOkIvv6G6deUxBLFARjJQ5KJEpDYTNIlJDY3lIIbDiyFG5JDJaDmmQ2mRjOujOXJ4jPNqFRoky1ZA0DTKFhMsKW9WC0IZzkYmcwqjrgSMyMH09p23B/6vXyKsPuZ3hzi3xrgpDucnupASq4uF1vhP0guBH5Rsuj34P3Q8mavolTsFztoRjjYlNM8abT8qgOOOOOPjGIQhCGMYhBDfoTcqsmMxgEZgjIWGhxtLUkR9Hwee11ykjlS7lhVC6WJpq9hcspObJibqZCycPOqL4MGpFRdLxUYnoCwCrzZxTvz5exeAwTGx1WGV3i0sFQbittMdORKtDFwPGNnAwKRJkqGc1isUlIQmSKKCINsULeU3LcwoZMOPDKh9scSJgYHHC/La0FdFxEQtUFsURenKCZxhUogKV2qBTMi7AdDBx/DXGFqmzbzPHEiZywYb2u8BJHrHpfRtPW4jDorEsW5OaikDfe3MfupJlqnTsr/SbTVqloNLbEwYBgiSpUDPM5nQ63X7NPcVTeZCSyMp3vVMTM1+Z6rvIMzBiJeHitPTJ7QfCfCeESS34ilAcAqloQtxiAwBxpNKa0mGnjU2by24S8Km0Y0gFdOZnqXm9mYkl1QizDhGeOYdZCHubrIxy+7Vm7K9e1j9ibK6JwCaHMNNkgk5IcJESNMQyWoeaZBXy8ltNUp4G0GxrPlmF4SILl+FRqxcgkHsYEk4gzhyIFDQ8YawllbegSgS85DrZbnTUVNxF8oFJrkJVhxlJlRZFUr+DGDIK1SrL4k7ic95PJqlOAVrq6rYWTHmU2gWXnLysqwVAwv5u3vSOwpitSSsVYYhZTIsAutArmJs41Sz3UB54GsGoJdHz2iQu0QQrSEMicmMrxpG1+DZCuwS8dQhURhIAlCXkkSYHh9X43imLAG78U6oHAyDigQdyr3U7Fm69X3zDHIyHN5iL7Jre0Y7lKArHRyvS6kmrKDI2LxBlUakqIAoMA1QiloTAOHAKIz5F5E1QlIv5UrNI7MSGHuCgdGeBtNjYVSaljlrmKMzoDVXVWxAmOQF9AQKKHSZwcI1CDKH1NkMkLSWNu3Zxb51Hey8gx8harBY1xy4CuVTd9F1RSrUVYFiQKlLYdlYoVc3VJuus0UksK/7bdaTMZHBJ3cCsrnilx4RYGdI1ymlPmhUvFqUzMo6iV0RMi/aNA6TQRzCgmBtWCg0aE7VjcYBU5FVXF49QK6Ufi90bcC5bovTPQFr01QNqrZQrFsF5QEoN9MVsEwDgWFvRpUOU0C4MncU5OwFoqXK+5wEKDD+Ut75SchiDiIUQAUAvQXMoR0jrqqJgClG3wm4jpTjkQBMWNJdIFJ/hbfCVZmstRLwU2/wS1zH1MhaAG078aBqGYDUAwI0CempxvagHxQOk6Ydnd5i7G9VNmTMiiwxMB7A5K/kThWgpHJNXs5MZZdv1b1s5kaG2auaWUIyGW0VjOdTmxoMuSpHerWUIhQ2JhFFHq1BB2kscKTCwptcRBoFvDBq60Mcy+YsFQumU4+Ag2LZQwIu/ZlSufGSleQdZfm9TAUJawaNQaxn36VtMYQKxii94R4ruCntM6wrshqwSbBLM1NSWergZNTBlE27ymhBMWPERacoWKjXEVV3CwCwa+3qCwaM5opYGAKleWtTQnQgiqAMTkKfs8MxNt3sugowkqNXCiHMuTGSaghXb8xznoGtBveZ0BSxdx3TFtkgjamwaOiUM1ismnaGEkoadpVCnvM2WWg7eLTM4qzKiRdXuIy41abu2o2BFGr0ZmbhaaBYzolLDDYNmvAUWDbxYlZGGNqpWZGA2bQtTbTVa7n6vrFJZ+MhqXOw+T52IiohEoQhxAUSggkBCEQiIUY1ZEABcCBVKUpTSqh8QVg8MJNmAQAOyyyKWlny2Mj92s/ba5VdcY9Cu8cU41tlMBIWqEkSgymw7qjkRUPDxMIAwYSlYoAAAAAAAAAABYAA0AAjFiyYUNAgAQQQQQREqNUtKKsAooa15Sn1oiIiNapaNa0NbDDC7MsRRVluxRF7lly50m2226qsEq5VSbbbuySl0KunOIjBllKk0WsWbDI22cVaLSlNLc7rWnhTWmJjGaqsNc1oCr7Org4p27OLECkJWl+7W8zZ07bqx2QlxzBRYON0rRUwYyDmg0DRNNVi3hR70c2NS2V1DCSWdpbGyfmb7FB+ALpmeFQF8YlIOjHG48PIMQyTGZysJhSquuQV0tPTFwF8ZoFhw5pBlexuvCFRqDPSDOxhvchjMEt80phOWbYxVZazLYJ8GlwiSz3AixaMbOhj1JmhZqFYoxK5Vlhky5XkCYRiqSiaKZ5QQZaG44877LXtcCIn4sUsUtwbs5M9R2KgrAQPym/sCiEQoRZvvbzOZrfN9n85GkjCrAmdeHNiUFykC7jHLka60+FpJGkr6EjEjbcapI069UlKxbi01ShkktyO9wwbVxVawlWVYjdqqlLLitYNlTtN+0Nk19JimsBHn6mHEPoxAkwYSDQBY0BnLNm0k6+yus36Xi5eZ98TNfH5fOxTf0Wzi36/b7zqv8K1/6/c9P/z/UFDRN6OCu4gB+q0gIwBqACEEEYQNAoAYBhIJBWAZgRuApYoKG2IgN0Iiiim6NG5Ciom9rcAZ4IOwo03YJRB96P2DJA2DTBsGmDYMX3IgNqDoQQKQ0A2khttA/cwI5toPKtCsGlrep/+LuSKcKEgreP0Kg=",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-21T18:46:31.565569Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-21_11_46_30-5442649581897125100'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0821160435'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-21T18:46:31.565569Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-21_11_46_30-5442649581897125100]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-21_11_46_30-5442649581897125100
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-21_11_46_30-5442649581897125100?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-21_11_46_30-5442649581897125100 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:34.255Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.067Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.106Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.174Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.209Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.308Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.353Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.386Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.418Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.453Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.501Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.525Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.560Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.597Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.636Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.658Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.690Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.728Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.764Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.795Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.821Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.855Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.893Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:35.929Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.042Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.079Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.112Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.583Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.653Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.689Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.700Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.726Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.737Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-a...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.801Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.801Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.872Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:36.907Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:37.003Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:46:53.849Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:47:04.822Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 2 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:47:04.874Z: JOB_MESSAGE_DETAILED: Resized **** pool to 2, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:47:10.196Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:48:39.268Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:48:39.298Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.614Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.686Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.725Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.806Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.841Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.859Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.877Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.895Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.927Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.932Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:55.963Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:51:56.001Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:54:16.047Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:54:16.117Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:54:16.169Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:54:16.242Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:53.563Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:53.631Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:53.681Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:53.745Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:56.035Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:56.092Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:56.165Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:56.198Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:56.217Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:56.255Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:56.288Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:56.327Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:56.397Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:59.204Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:59.266Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:59.337Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:59.383Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:59.449Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:56:59.517Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:57:02.312Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:57:02.394Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:57:02.507Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:57:02.593Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:57:02.624Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:57:57.621Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:57:57.662Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T18:57:57.690Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-21_11_46_30-5442649581897125100 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598035587502/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05732369422912598 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: dfcd7be18c454aa48773445f85794daf and timestamp: 1598036372.7135894:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 784
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598035587502/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598035587502/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.056829214096069336 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 785.735s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 8.396 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 15s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/gqc3ss4c5z47k
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1828
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1828/display/redirect>
Changes:
------------------------------------------
[...truncated 290.98 KB...]
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWXosx98ACKN/////////////////////////////////////////////4AjT6gN26rgjZtCxsgC9g1E4g0UIZKYniNI9TTEeo9JtJ6mj1MmRkNANNA09QAaAAGgAAA0ABoAaAAAA0ANDQAaABoHqGhoNNDQaDTaQZIRpMTEwQCaak9I9Q0epptQAAANA0aZAAAAAAZAAAAAAAANAAAAAAAAAAAAABkjYoAAAAEwCYAEYATIwAAmACYABBgTEYBMJkwAAAAAAAAEwJoyMCDAAACaEHqf6pBiEwAjTE0wATBGmTARppkwBMARgAmACMAEaaYmAEwAAI0wACYAAJpgACDCGjQMACTAATAAEwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEiRBohMIIyNU9oU/QgjQeKB6maj1MmGoxAADQNNNAGQepmmpoAAAA0NADQAAAAAAAAAAAAAGVTG2N0MIEsmXFZl2KiN8TzPiaGvXsYUOCY7BdZBUWLq48k7FE/bttdcZRUXKrQTanofeppgXO/AKhjwLpqhEICFCgQEjTEm0gGamid+mkuXTGNd16B6pKbGNCUxCCdbyKAMWd4CwFAniAjQLAeC+EEVytfnu3r1nAgtW7maamEoahhMRnHjf8GmkeINAInAtVmmbI2JqnFZNQNIpeXHSkUFvIzznyZgOlPDkgtMQYu/lh4m8MVxMsRZjJB4CvcZiwLJ1kwmE4JyFKhKYTAnaT8hNTlARwvAxw+i6NtznOSlZxIsTc2cSRpWwzDwqiU64Rff0N07MpiCWKAjGShyUSJSGwmaZKSGxvKQQ2HFkKNsSGS0XNMhtMjGdgGcuTxGebAKjSJlFkDRNQrLSZaV060FAhnOxiZzCqOUCRmRg+nodwwf+rV5i676211eIx/cR8n1+J2EgJdgW86Pu/VCLdSOQcsJGAEBzO/lb0Jc65cLUEY4R7JaYm52hpakMMMMMWCCBBBBBCCCBBEie8cEIhKAhgCiOACA9+n1BJSYoo/Sb/n9ZcrI5Yu4IqhdPE01ewuWUnNkxN1MhZOHnVF4GppxUXS0qMLahYBLzRy5314t3/CNVeWstZiFzgF27RiWN2rgqOrGzMClQeLopyzPCs9ZVI0U7x9hTgbObg2WT3DcQoZIONGUDz8UKJhYXGi+X5aldFwkQPqC2KIwDkxM4wqUQFdlFApmRdjOhg4/hrjCimzZzPHkiZwQw3td6CSPRPQePXeh2GDRWJYd3eFFG1660x+WyZaVf1GPkNvplS4GlsCZswzRJUqBnISGc1ElrU9lUNtjOPjZbs+eY9kz952GfAZmDFYZMqq+tX5B/J/JkiSWjSUUNWalCsiNlC0GrJO0FFAkCkq3U5CwanlxqqLNftBX+VwJXT6CkbtlELY3vTDzJxNweeFHMbnVLirknEjqza3FOCTQ5BptUEndDeIjhUZDHaN8o0HeLsWu0qnbcwjSqeCYYhIgwY4FGpFwyQfhfiTijOJIgURExprCWVZ9BKBLzsOxlM6aipuIvlApNchKwOMrmVFsVSv38YMgqKVhfEncTnuZ5NUr3xYutqxhbMeZTaBZedvLCrBUDC/nKd7R2RMVEkrVYGIWUyLALrwK5ibONUs91Aa0GUPoJNDzmiMu0VIFpAGQuTGRppG0+BZCuvS06hCoTAQBKExJIkwO55/ttlMOANr3ZxQLTIOyBB7Ktc/p2LpQ24Z4pGT5PIRpUou6pjr0QCp83IyvPlFZQhG0ZQhUymqiAKEANUItcEwDiQCiM+ReRNVkpF/LFhpnaSQw9wVh0h4Ohr66qTUsctYwxGdAay6y2IEx6AvICBRQ6TODhmqQZA8p+MxwPksbdN3i3jqO+l5Bj5CitFjXHLfK5VN31rrClYoqwLEgVddIdtgoVk3VJuyw0kksK/7S6hMxkb8ndvrCyeKXHhFoZ0jWK9OfNCrvFsq8zK3USuiJkX6DQOuaCOYUEwNgwUGlWTosbjAKvkVVcXj2QWV1vxm3NmBduUYBnoC16SoG3WsoVi4C1QEgODMVsEwDgWFtiSl8VuGAavYpCdcLO0WX9a0Q4MQJeX2y03EEHYQ4ggoBdwwawq+OuSomYKWc+6aipWbdyAJo0oa+nJQMPbIWpNllnJiCmoGDXPA+ZkLIA5nPfPOA2AcAGBZgJ2Zm3VnAf1AvnDD09vkZX32ntibEUYGhgPwHJX8icCgKRyTV7OTGW3b1U6+cyNDbNXNLKEZDLQVjNdPmxwMmq0jvFrKEQIbEwhhi1Kgg7OWOFJhYU/XEIfAuYgNbWBj2YLFgqF4yHIVCDYuFDURd9mUK58ZIV5F1l84CYCsltA0qg2hn36dK8YQK1ii94R4rt+noZ1pZbDVok2CWZqaks9XAyamDK027yvRgmLHiIocqWqtriKq7gYBYNbZ1BaNGc0V2hgCpXlGpoTrIIqgDE5Cn7PDMTbd7LoK2ElW1cKIcy5MZJqCFdvTHOeibSDdczoili7pt2LYpBGwNc0tIrM1qtmnQMJJQ06FUKe5zZZaLpxaZnFWZVpF1m1jLjVDcUqNcRW1ejMzarUQLGdGpYYbBs1oCtYNnFqVsYY2ClbkYDZoFE201Yub6brlcs/GQ1LnofJ89ERVbbykpS20tj01s7rbbcM7aVVY9i3TM86lKUVMVk5aEVSt4Tm27JJGpaWfKYyL3JXrj7lZ1xjYF2nFOPv2UwkgfUI4VBlNh3UHISoeHiYQBgwlKxQAAAAAAAAAACwABoABGLFkwoaBAAgggggjeG6JPQyUNDQuusiLZNtt3XRJ3XXF11999eaRgwZZV4Ii9yy5c6TbbbdVWCVkqpNtt3ZJS6FXTnERgyylXNFGLNhkbHOKtJpSmlttvtJ4U1qCYxmqrTWNpAVfZ1cHFOm7xYgUhKhfuFuc2dOl1g7YS45gotHG3VBV4MZBzQaJpGorBbkre6HNjUt2uqYSSztPX3Z+ZvssH4AvGZoqgvDFpB0I47IBqDFMcxmclEwpVXXGK6W2zFwF4ZoFhw5pBlevt/ClRshnpRnZg3WQxmCW8aUwnLNr4qsrRK8CmkOvghZnRQIeNdWeUyXGCl64GgqiL5aHhWllagTDMNWQTQzNUEGVguuNd9h9utBET8GKWKX8benJnquw0FYCB6Q37AogECEWcH2sxmJfl+v+dTpIw5QTOvEmxaC5SBexjlyNZanA00jTV9ZIxI2PGqSNSzVJStW1oapWZJLbDvcMG1cVUYSsKsRuFVXXbcWLBu1OhvWhsmvpMU1gI81Tw7ClDCCUBhINwFjcGflcueO8+mstoDzRnm9ntrEz7/h8a9NfJdNrvn9PrOK/srY/j8zv6/f9nv9PuqCDuggh+KqgI0BwACEGEYUNwoAYBhIJBWAZgWa+0UT3+tKIDpCKn4t0jQO4n591aFoM48PQoE3oJRB96P2DJA2DTBsGmDYMX3IgNgDrIIFIaAbSQ22gfuYEc40HlmhWjS2nqv/i7kinChIPRZj74A==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-21T12:46:32.906118Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-21_05_46_31-1794211109423176823'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0821100314'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-21T12:46:32.906118Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-21_05_46_31-1794211109423176823]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-21_05_46_31-1794211109423176823
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-21_05_46_31-1794211109423176823?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-21_05_46_31-1794211109423176823 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:35.579Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.300Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.429Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.500Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.529Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.642Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.685Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.725Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.759Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.785Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.823Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.853Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.895Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.918Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.952Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:36.989Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.021Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.055Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.084Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.121Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.153Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.186Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.220Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.259Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.295Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.329Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.368Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.584Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.647Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.675Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.697Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.703Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.730Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.793Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.793Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.872Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.900Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:37.933Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:46:49.246Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:47:03.787Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 6 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:47:03.821Z: JOB_MESSAGE_DETAILED: Resized **** pool to 6, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:47:09.054Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 9 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:47:09.080Z: JOB_MESSAGE_DETAILED: Resized **** pool to 9, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:47:14.324Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:48:42.634Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:48:42.673Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.073Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.145Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.189Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.257Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.280Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.306Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.317Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.327Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.354Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.392Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.419Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:52:01.444Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:54:15.718Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:54:15.807Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:54:15.868Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:54:15.950Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:56:56.948Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:56:57.041Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:56:57.091Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:56:57.165Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:56:59.807Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:56:59.917Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:00.004Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:00.043Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:00.063Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:00.097Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:00.136Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:00.179Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:00.253Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:03.019Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:03.097Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:03.171Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:03.332Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:03.471Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:03.599Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:07.006Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:07.309Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:07.586Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:07.724Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:07.774Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:56.967Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:57.011Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T12:57:57.072Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-21_05_46_31-1794211109423176823 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598013989231/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06664705276489258 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: f9fe3fee02d04e5e9bfdd117b4fe8633 and timestamp: 1598014789.5929508:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 800
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1598013989231/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1598013989231/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05712437629699707 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 800.870s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 23.435 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 32s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/g63grghsq2oj4
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1827
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1827/display/redirect>
Changes:
------------------------------------------
[...truncated 288.54 KB...]
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWYY92SgACKN/////////////////////////////////////////////4AjT6gN26rnTHbNCxsgHb3BqJ4g0kKNlJiZo0mExHqMm0nqZNGEwjJpo00DT0QBk0aGgaBoAAGgGhoGgAA0DQMQaADQGgAaD0JgI0yMjEMnpBkhGkxMTBNIyaFPKPUNMmm1AAAA0DRpkAAAAA0DQAAAAAAADQAAAAAAAAAANAABjU8SeUk9BGjT1GmmCZMjJiBpmk0yaPSeoyeoYQ0BiaANNAxGIGmjEMJiaeowE9RpoZBgRo00GmjIMTQ0YRhMgDQ0ZMmhB6n+qQYhMAI0xNMAEwRpkwEaaZMATAEYAJgAjABGmmJgBMAACNMAAmAACaYAAgwho0DAAkwAEwABMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABIkTQITCCMjVPamT0Jqeo0ep6agepmo9TJpk0xAADQNNNA0ZBo8p6mgGgAAA0AAADQAAADQAAAAAABlUw6t1MIEsmXFhlyKiN8TzPiaGzZqwob8x1F1kFZYuLjyTsUT97etdcZRWXKrQjeT0PvU0wLnfgFYx31w1wiEBChQICRpiTaQDNTRO+zSXLpjGu6+EeqSmxjQlMQgnXMigDFneQsBQJ4kI0CoPBfCiK3YwJ7l27avoLy9t5tqYS01DCYjOOj/waaR4k0AicC1WaZsjYmqcZk1A0i28uOkItFx0Z50BMwHSHiCQWGIMXfis8VeGK4mVorxkg7/Tb5isK50JhMJwTkKVpKYTAnYT8hNTlAR0PgI4fheibc5zkpV8SK03NnEkaVkMw9DUSnSEX38K6deUxBLFARjJQ5KJEpDYTNMlJDY3lIIbDdEKNqSGS0XNMhtMjGcsM5cniM82AVGkTLVkDRNQoWEywpb1kLQhnOxiZzCqOvhIzIwdNodwwf+gYyDutAri2OEb1kbK+PoeRACbTlKTNJW2IuTI6GcTyLiHHseuPxgmzQUTgAfEGwKflSkso53ROOOOOPjGIQhCGMYhBDfoTcnhDMkYBGYIzb7h/G2tyRH6DgdBrLlJHKl3YBVC4sTTV7C5ZSc2TE3UyFk4edUXga2nlRdLSowtrFgFjmzlzvrxbv+Ea63FY0rtcCyowqzVNR0XKqN8WbM4SiY44eWee8S0FnqRpJ3j7CnA2c3Bs8nuO5BQyQcaMoHn4oUTDw+OGCX5a1lFwkQPqC2KIunJiZxhUogKV2qBTMi7GcKDj+Gt2WqbOJM8eSJnLBhva72ED5k2w9++nWERx6ZDMsTGkk643xT9WrRLasvgUfA1ukl5kBkOASlSFJ5Ly8DGj4+Q2+53qPCtsnnunn5ryrZ+ho/zWPLYDGMDCAamp+bVTcc3k3k1HkhHCKUBwCqSYOfMQGAONI7Gtphp43k5dphUqfsphpAKygxxo+nSyMWt9BE1MEL6p9mlX2KZpaJURb3h1/dLNjvDKsMi5onTBkVRyDEdKHiWaztBtGxrdIzElv5JLzzlvcHv2T6gIEE6jKnvBS5HNwyBIv5jPpECfH2FMmJNcW0EkEPPQ62W501FTcRfKBSa5CVYdPpMqLIqlfwIwZBWqVZfEncTnuJ5NUpvytdaVbCyY8ym0Cy87eVlWCoGF/OW88jsiYrUkrFWGIWUyLALroK5ibN2pZ7qA1oMofQSaJjaIy7RWgWkAZC5MZGmkbV4NoKzCXjqEKhMBAEkSgkCRA9C2q4yWYAEn6alIMBiDCcQaCge53713nnkKc2fLdSYltfe4/ks+XpB1X4HD+D8P31FvE+L2O8W33VSBbwHUcljLAfWAURnyLyJqhKRfypWaZ2kkMOlKB6E8Faa+uqk1LHLWMMRnQGwuwtiBMegLyAgUUOkzg4hrkGQPKfjMcD5LG3bvcXHOo54vIMfIWqwWNcat+rlU3fRdXUq1FWBYkCpS2HZWKFXN1SbrrNJJLCv+23WkzGRwCd2/rK54pceEWBnSNYppz5oVLxbKmZlHUSuiJkX6DQOk0EcwoJgbBgoNKhO1Y3GAVORVVxePZBXSj6PbGzAuW6L4z0Ba9KUDe2LSFYtgtUBIDfzFbBMA4FhbYktPSdgnCpoFmRgikGmboGAPQMHIvjRTVO+EGEPQcwkBignVO7LERS8SkFNTlJTP/bp2IgCcHCxstsQcnrBO7wrTQ5KAVM5OjIxioMQvgE5U0tcsCsBYAYCGArKOnXQ4DekGWpMGdobIS7trqrJWRQgcDANwHJX8icG0FI5Jq9nJjLLt8revHMjQ2zVzSyhGQy2iqzfUJsYDJq1I75ayhECGqYQwxanQQdoLHCkwqU/ZEIfAt44GxrIx7L9iwVC6ZDkKxBqtlDWReAzKFc+DIV5F1mAb5MBQltA0qg2hn36dtMYQKxii94R4ruAnoZ1hXZDVgk2CWZqaks9XAyamDKJt3lNGCYseIi05QsVGuIqruDgFg1tnUFg0ZzRSwMAVK8tamhOhBFUAYnIU/bYZibbvZdBRhJUauFEOZcmMk1BCu3xjnPRNpBueZ0RSxd02zFsUgjYGuaWkUM1ismnaGEkoadpVCnuM2WWi7d0mZxVmVEi6vioy41abe2o1xFGr0ZmcUtRAsZ0SlhhsGzWgKLBs4sSsjDGwUrMjAbNAtTbTVa5v1/W6Sz8ZDUufh8nz8RFRCJQhCvBRTZBICEIhEQoxqqIAC4ICoUpSmlUz4g4M8MJN0AQAOySRqWlrzGMi92V62+5YdcY2Bd44px9+0mEkD6hHCoMpqd1FyEqHh4mEAYMJSsUAAAAAAAAAAAsAAaAARixZMKGgQAIIIIII3huiT0MlpoaF11cRZJttu66JO664uuvvvpmkYMGWVMERe5ZcudJtttuqrBKuVUm227skpcJXTnERgyylSaLWLNhkbHOKtJpSmltdttJ4U1qCYxmqrDWNpAVfZ1cHGO3e4sQKQlaX7dbjNnTturHZCXGsFFg42ytFTBjIOaDRNI1FWLcFHuRzY1LereYSSztPX3p+ZvssH4AumZorgvDFpB0M4zIBqDFMcxmclEwpVZXGLKXrZi4C8M0Cw4c0gyvX23hio2Qz0wzswbnIYzBLjmlMJyza+KrK0SvAppLr4IWZ0UCHjZVntMltgpethoKoi+Wh4WJZWoExDDXkE0MzVBBlYLjjXfafbrYRE/Filil/Duzkz13YaCsBA9IcCoUQCBCLV/7nZZiX5vs/nVKSMSUEzrx02LQXKQLuC5cjWWpwdNI01fQkYkbHdqSNSvVJSsXFWmqUMkltR3uGDauKrWEqyrEbdVUpZcVrBvVO03zQ2TXTMU0wI2rdQwi0COEuDBAOwCZ2BjfKdnZhmnrmrcVsItoaKAo6X6fW7U32ZKdkqPv+Klfql9R+qusrf3/a6v/y6wdMQHMVb8cIUBYACDoD3cOwSAMAYIBAL4BjAQ2PaRXV9/PAXHP/+g1x9gxBcusL8wAxo0M4nkWcJRB96P2DJA2DTBsGmDYMX3IgNgDoQQKQ0A2khttA+lgRzjQeWaFYNLaer/+LuSKcKEhDHuyUA=",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-21T06:46:46.571952Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-20_23_46_45-1786305574955187248'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0821050323'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-21T06:46:46.571952Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-20_23_46_45-1786305574955187248]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-20_23_46_45-1786305574955187248
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-20_23_46_45-1786305574955187248?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-20_23_46_45-1786305574955187248 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:49.235Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-f.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:49.909Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.020Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.112Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.159Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.264Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.416Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.483Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.588Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.638Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.678Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.708Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.748Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.777Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.810Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.846Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.881Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.915Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.946Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:50.977Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.014Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.048Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.081Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.115Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.135Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.168Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.204Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.481Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.553Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.602Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.613Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.631Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.641Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-f...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.699Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.699Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.758Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.789Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:46:51.827Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:47:27.898Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 6 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:47:27.972Z: JOB_MESSAGE_DETAILED: Resized **** pool to 6, though goal was 10. This could be a quota issue.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:47:29.692Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:47:33.311Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:49:01.931Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:49:01.970Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.394Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.458Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.497Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.573Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.608Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.643Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.666Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.690Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.700Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.759Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.784Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:52:25.822Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:54:43.238Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:54:43.305Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:54:43.359Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:54:43.426Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:27.974Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:28.121Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:28.169Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:28.234Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:30.506Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:30.575Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:30.633Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:30.668Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:30.692Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:30.722Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:30.750Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:30.782Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:30.844Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:33.628Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:33.699Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:33.760Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:33.813Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:33.879Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:33.954Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:36.553Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:36.624Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:36.701Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:36.842Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:57:36.880Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:58:29.493Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:58:29.526Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T06:58:29.557Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-20_23_46_45-1786305574955187248 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1597992396402/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.13533401489257812 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: acda293da6db4f839d9774670dbcf8ea and timestamp: 1597993208.6965258:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 811
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597992396402/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1597992396402/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.05869603157043457 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 812.832s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 37.486 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 14m 51s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/ujbrsyb4nqmeq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PerformanceTests_WordCountIT_Py37
#1826
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/1826/display/redirect?page=changes>
Changes:
[daniel.o.programmer] [BEAM-3301] Consolidating starcgen method names in graph/fn.go and
[daniel.o.programmer] [BEAM-10289] Fixing miscalculated SDF split fraction.
[noreply] [BEAM-10701] Generate Codecov XML (#12566)
[noreply] [BEAM-9979] Fix read index being reported to be reset after any metrics
[noreply] [BEAM-9546] DataframeTransform can now consume a schema-aware
------------------------------------------
[...truncated 289.82 KB...]
"python_side_input2-Write/Write/WriteImpl/FinalizeWrite": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "SideInput-s20"
}
},
"output_info": [
{
"encoding": {
"@type": "kind:windowed_value",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "FastPrimitivesCoder$QlpoOTFBWSZTWYQR6NMAAEDXwH8QgCEJAEBAv279AmAAIABqEqnqGgaABpoyNAGVHpGgMgDQBk0oR6IeoECBiqqU5NY23ndshzT2UPUOGrg42YPi9VyA8lbwwPJgtghxs5Qq1aWwExCDeMa0RHC2QigTCdizz1nx+LuSKcKEhCCPRpg=",
"component_encodings": [],
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
}
],
"is_pair_like": true,
"pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_5"
},
{
"@type": "kind:global_window"
}
],
"is_wrapper": true
},
"output_name": "None",
"user_name": "Write/Write/WriteImpl/FinalizeWrite.out"
}
],
"parallel_input": {
"@type": "OutputReference",
"output_name": "out",
"step_name": "s7"
},
"serialized_fn": "QlpoOTFBWSZTWYPO1NwACKN/////////////////////////////////////////////4AjT6gNtVYI2zQsbIBPYNROINFCGSmJsjSNNMR6jJtJ6nqNGjTENANNBk9TQAaNAABoAAGgAaAAAADQMgaAABoAGgwmhoNMjQyDGkGSCaBGRplMjJoRqPUNGmm1APUGgaDQGj1PUAAAADQabU0AAAA0AABo0AAAAAAAAAAAAAJMABMAATAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABB6n+qQYhMAI0xNMAEwRpkwEaaZMATAEYAJgAjABGmmJgBMAACNMAAmAACaYAAgwho0DAAkwAEwABMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABIkQCAIajI1T8hPQmp6jR6niQeo8o9TammE9EaAA0A00DRkGmYpoAAAADQAAANAAAANAAAAAAAGIl/X3kxgVZcUVGWqc8kIrTQiaO4uK+NjvDJXLzIoFi0uTXHqdOG6unPNZPAuqrSDc1p4X6KIl1oYhAZLu0aQRCAhQoEBI0xJtIBmloHfppLhpjGu69E9MlNjGhKYhBOwZRAGLOWCwFAnigjgK4fC+GFOzUva1q3bsXcVzdWc70r5YaRfMBhPG/4NFI8QZ4ROBabNE2BrzTOPx6QZiy4tOlIsFvIzjoCZeOlPDkgrMAYO/lR4m4MFpMqRViJB4Cna5SoKp0kwmE4JyFKwlMJgTrJ+QmpygI6HwMcTi9I25znJSq40VJubONIzVwy/0NBKdMIuu6O2dWQwBLBARiJQ5KJEpDYTNElJDY3kIIbDcEKNmSGS0HNMhtMjEcuM4ZPAZxrgoMxMsWMNA0ikrJlZTZ1oLAhnOxgZzCoOwhIyovdPn9wvf+sd/hbN7md7cIZ5a4WQ7vJ7yQEovKP6/x/dKLgx2Wll8c9h76PljcAJQ7LeLiEbud/zXO9B0XTdVm3nnnnnNaxjGMa1rGLX/tI6LQNECYQYN8y2+t0zJZPC9BwOg1VykjlS3lhUC44TTVzC1ZCc2TE3QyFj4mGgvgwZMqLxcVGN0BYBUzRxTl3y7GFka2qz8rxV0FMymZuY0lyyjkTYnEVdLo4rwl6M54FnrKpGZO+PhpwNnd4Njh7tu4UMqGuGIH4acaJf3/Hi+MNWCwi4yIoVBbFEWzhiZrChRAU1WKBTMa5o4kG/4q3JYps2czx5ImcuF+5rvgSR656b07WXbhGIrEsc3uCijTA9yZn6qmWrXdph6jf9YqcQaW3JixDFElSoGcbG6PXbDaJ7msb3J8zKSfflZn3zX5oK47AzMGLC6uted2Twg8k8k6oklqklFDVmpSodFqFoNWSdwKEBIFsbJXY207DoW1VCmu3YtHO43T10dUOE6wS2I8ky5SEHFOUjFMIuBi1exa6fZmyik31NCSNNkgk3oR6IkachldU7U5pgb4W21inhlq09QPmmWASIL2GJRrZcgkHgYQk4QzgyIFBQcQawllfdwSgS89DqZZhmoobiLpQKTW/lUHUaZlBXFEruBF7GKxSqLok7Sc9tPHplO+Kl1tVMK5jyqbQLJztxUUXqAYXc5ZzyOaTFYklWqgwCyGNXhdfBWsTZuVLOdAHOBqiFBLhaBPIXp4IlpEGRtmMrjQNu8CwFZxLl5CFRmIgCUJgSRJgeKV/G+UxoA3vjnlAtMg3IEHQrH1eza+xXOAxyCMjzZKWz97i+Sz5eYOR+BxPg/D99St0nx2v3S2u4oQLdA6DhYisH1kFEZ0i4iapJSLuVKjRO0khh0xSHoTwdhrayoTUsUtUx0zWQGouotiBMggL6AgTzvE1gcA0iDKH1QyGSKElibs3eDeOg54uIMW/sVYsS5Bb5Wqhu6ldYUqlFF5YECppsh11ChVTdEm6qjMklfX/bLbCZiI4BO3fVFU8EuuBFYYZGqU6M+bFTcLYU5WUuglbETIuz2gdM0EcwoJga5goM1JOxYnF4VPJKi0uHsAqppfjtobEC1ZnuzWnC16coG6qWEKxZBcoCUG8mK2CYBoWFuppU+W/BeGToKgnZS0dNlPctEADD4Xr+EnOQRBuEAH0KAX0L2T+uu95VExBSct8ZuE6k43kAT09UtdVknyAub/W2ayJJgCm3xo4uAnTIQ4EtPfKhahmA1AMC3AUE1ONokB5UC6eMObo7Q1od6uzJmRQwejAeAOFdyRwbAUjhNXM5MZXbvVZ2A5kaG2aeWWQIxmSwVfPNTmxwMus0DvVrKERIa8wjjp6tOQdsLGlJhXVDcCMQgWceDUyoyDLxiwVC2ZTkYCDXslDAi71mvlda+lK8o8y9N2l4pJagZqA1DOu0bKcQQKtii53x4LeAnn4ayquGqxJsEsrU1JZytBk1MGUpt3FOhBMWLARYcoVqlrjKi3g3hXtXY0BWNGFoprC8FCuLGpoTpIIogDA5Cn7a/MTbdzLYKWElS1aKIcy1MZJqCFbvTFOegakG35nQFLB3TaMWvSCNcaxmzFJlrVc07AvklDTsKIU9tlyS0HZuEzCUZVSkW1bKMmJWG1soNYRS1cjKzZLSQLEdIpX4bBs1YClXtjFaVcX41ylXjYDZnlibaaqXdvXddplnYiGpc/D5Pn4iKH39VWteMXYzb7IX336TH4GtlsIkEySQQQQGCm5i51NW/l0YiPSyyKWljy2Mp+5VfswtqPNY6Jdy1TYYbCY1IoVCSNQZRXe1RsZUPDxMIAwYSlYoAAAAAAAAAABYAA0AAjFiyYUNAgAQQQQQRu/bEnn47DPz7baoiuTbbdtsSdttpbbdddTlkXr2SVN6IucsmTDJtttuii9KqVEm227ccpdGrZziIvZJSpmixiy35GvwlGZpSmls9pqTvprSExjNNVmqakBR9nTvce7N3gwApCVhdtVtsuGdltQ64S5BgorHG0Vgqb2Ig5sNAzGkqhbYpe3HNjUt2t1fJLDo627PzN9lg/AFszOFIF8YNAOknHZEOQYJkmMzaomFCrhcguEunTFoL4ywK/fyyDI9baeFKDYDPTDOzBt8ZiL0t40phOWXWwUZHCq+CmSehBCzWRQIfNwrQ6JbLBQ/ZDgVUzCXB8VKtVyBMAx0pRNHM5QQYkVprnvZhdhhET8GKWKIb+3WJrUno5ysBA/Kb2uJ4hEhFi89vsM5V+X6/51egjAqgmsvHzYM5bQBbvm2o1VpcHRSNFXUkjAjX7lSRpVaZKVa2VhplJjktmO5wwbVpRYwlUUYDaqimmu0qV7dqdhvWhsmunYprAR6CrhuFSEYOgDCQfgFj8DP2S8vJOUxYmb3Kw0r7/hYZr5fP6Web+rjOOM79vvPK/wrYfqfoP3/P9Q0VG2pGkWH0T9eQEQA1ABBoEX8PwUAMAwkEgrAMwLdg6aKGiuCIDZCEo4VsjSN4bNqS3rQZ4UOYmE3MJRB96P2DJA2DTBsGmDYMX3IgNcDpIIFIaAbSQ22gfTQI5xoPLNCrGlqer/+LuSKcKEhB52puA==",
"user_name": "Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
}
}
],
"type": "JOB_TYPE_BATCH"
}
apache_beam.runners.dataflow.internal.apiclient: INFO: Create job: <Job
createTime: '2020-08-21T00:46:38.521202Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: '2020-08-20_17_46_37-12002284678593248843'
location: 'us-central1'
name: 'performance-tests-wordcount-python37-batch-1gb0821000641'
projectId: 'apache-beam-testing'
stageStates: []
startTime: '2020-08-21T00:46:38.521202Z'
steps: []
tempFiles: []
type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
apache_beam.runners.dataflow.internal.apiclient: INFO: Created job with id: [2020-08-20_17_46_37-12002284678593248843]
apache_beam.runners.dataflow.internal.apiclient: INFO: Submitted job: 2020-08-20_17_46_37-12002284678593248843
apache_beam.runners.dataflow.internal.apiclient: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-20_17_46_37-12002284678593248843?project=apache-beam-testing
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-20_17_46_37-12002284678593248843 is in state JOB_STATE_RUNNING
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:40.822Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-1 in us-central1-a.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:41.828Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:41.867Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Write/Write/WriteImpl/GroupByKey: GroupByKey not followed by a combiner.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:41.933Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:41.966Z: JOB_MESSAGE_DETAILED: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.061Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.121Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.157Z: JOB_MESSAGE_DETAILED: Fusing consumer Split into Read/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.191Z: JOB_MESSAGE_DETAILED: Fusing consumer PairWIthOne into Split
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.215Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial into PairWIthOne
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.250Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Reify into GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.284Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/GroupByKey/Write into GroupAndSum/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.319Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine into GroupAndSum/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.348Z: JOB_MESSAGE_DETAILED: Fusing consumer GroupAndSum/Combine/Extract into GroupAndSum/Combine
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.383Z: JOB_MESSAGE_DETAILED: Fusing consumer Format into GroupAndSum/Combine/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.420Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WindowInto(WindowIntoFn) into Format
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.459Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/WriteBundles/WriteBundles into Write/Write/WriteImpl/WindowInto(WindowIntoFn)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.542Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Pair into Write/Write/WriteImpl/WriteBundles/WriteBundles
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.598Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Reify into Write/Write/WriteImpl/Pair
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.646Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/Write into Write/Write/WriteImpl/GroupByKey/Reify
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.725Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/GroupByKey/GroupByWindow into Write/Write/WriteImpl/GroupByKey/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.783Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/Extract into Write/Write/WriteImpl/GroupByKey/GroupByWindow
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.848Z: JOB_MESSAGE_DETAILED: Fusing consumer Write/Write/WriteImpl/InitializeWrite into Write/Write/WriteImpl/DoOnce/Read
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.916Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:42.965Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.003Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.025Z: JOB_MESSAGE_DEBUG: Assigning stage ids.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.250Z: JOB_MESSAGE_DEBUG: Executing wait step start35
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.318Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.354Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.365Z: JOB_MESSAGE_DEBUG: Starting **** pool setup.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.388Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.389Z: JOB_MESSAGE_BASIC: Starting 10 ****s in us-central1-a...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.446Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.446Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Create
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.515Z: JOB_MESSAGE_DEBUG: Value "GroupAndSum/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.539Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/GroupByKey/Session" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:46:43.587Z: JOB_MESSAGE_BASIC: Executing operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:47:11.246Z: JOB_MESSAGE_WARNING: Your project already contains 100 Dataflow-created metric descriptors and Stackdriver will not create new Dataflow custom metrics for this job. Each unique user-defined metric name (independent of the DoFn in which it is defined) produces a new metric descriptor. To delete old / unused metric descriptors see https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:47:11.520Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of ****s to 10 based on the rate of progress in the currently running stage(s).
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:48:48.431Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:48:48.507Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.417Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/DoOnce/Read+Write/Write/WriteImpl/InitializeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.477Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/DoOnce/Read.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.515Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/InitializeWrite.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.600Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.622Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.651Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.661Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.670Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.706Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.730Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/WriteBundles/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.766Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:52:03.813Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(InitializeWrite.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:54:47.689Z: JOB_MESSAGE_BASIC: Finished operation Read/Read+Split+PairWIthOne+GroupAndSum/GroupByKey+GroupAndSum/Combine/Partial+GroupAndSum/GroupByKey/Reify+GroupAndSum/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:54:47.870Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:54:48.085Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:54:48.324Z: JOB_MESSAGE_BASIC: Executing operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:29.626Z: JOB_MESSAGE_BASIC: Finished operation GroupAndSum/GroupByKey/Read+GroupAndSum/Combine+GroupAndSum/Combine/Extract+Format+Write/Write/WriteImpl/WindowInto(WindowIntoFn)+Write/Write/WriteImpl/WriteBundles/WriteBundles+Write/Write/WriteImpl/Pair+Write/Write/WriteImpl/GroupByKey/Reify+Write/Write/WriteImpl/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:29.727Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:29.779Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Close
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:29.833Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:33.003Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/GroupByKey/Read+Write/Write/WriteImpl/GroupByKey/GroupByWindow+Write/Write/WriteImpl/Extract
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:33.073Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/Extract.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:33.142Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:33.176Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:33.195Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:33.224Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:33.265Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:33.305Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize/_UnpickledSideInput(Extract.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:33.365Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:35.779Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/PreFinalize/PreFinalize
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:35.847Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/PreFinalize.out" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:35.927Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:35.980Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:36.066Z: JOB_MESSAGE_DEBUG: Value "Write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output" materialized.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:36.258Z: JOB_MESSAGE_BASIC: Executing operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:38.557Z: JOB_MESSAGE_BASIC: Finished operation Write/Write/WriteImpl/FinalizeWrite/FinalizeWrite
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:38.624Z: JOB_MESSAGE_DEBUG: Executing success step success33
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:38.742Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:38.826Z: JOB_MESSAGE_DEBUG: Starting **** pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:57:38.862Z: JOB_MESSAGE_BASIC: Stopping **** pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:58:33.912Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 10 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:58:33.953Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-08-21T00:58:34Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-08-20_17_46_37-12002284678593248843 is in state JOB_STATE_DONE
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results*-of-*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1597970794852/results[^/\\\\]*\\-of\\-[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.06404757499694824 seconds.
apache_beam.testing.pipeline_verifiers: INFO: Find 30 files in gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results*-of-*:
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00000-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00001-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00002-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00003-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00004-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00005-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00006-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00007-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00008-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00009-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00010-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00011-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00012-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00013-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00014-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00015-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00016-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00017-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00018-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00019-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00020-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00021-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00022-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00023-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00024-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00025-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00026-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00027-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00028-of-00030
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results-00029-of-00030
apache_beam.testing.pipeline_verifiers: INFO: Read from given path gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results*-of-*, 26186927 lines, checksum: ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710.
google.auth._default: DEBUG: Checking None for explicit credentials as part of auth process...
google.auth._default: DEBUG: Checking Cloud SDK credentials as part of auth process...
google.auth._default: DEBUG: Cloud SDK credentials not found on disk; not using them
google.auth._default: DEBUG: Checking for App Engine runtime as part of auth process...
google.auth._default: DEBUG: No App Engine library was found so cannot authentication via App Engine Identity Credentials.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 221
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance HTTP/1.1" 200 None
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Load test results for test: b42d9d4ada5b4c199cb541160b5e70e6 and timestamp: 1597971614.8037217:
apache_beam.testing.load_tests.load_test_metrics_utils: INFO: Metric: runtime Value: 819
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/datasets/beam_performance/tables/wordcount_py37_pkb_results/insertAll HTTP/1.1" 200 None
apache_beam.testing.load_tests.load_test_metrics_utils: ERROR: no such field.
apache_beam.io.filesystem: DEBUG: Listing files in 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results'
apache_beam.io.filesystem: DEBUG: translate_pattern: 'gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/1597970794852/results*' -> 'gs://temp\\-storage\\-for\\-end\\-to\\-end\\-tests/py\\-it\\-cloud/output/1597970794852/results[^/\\\\]*'
apache_beam.io.gcp.gcsio: INFO: Starting the size estimation of the input
apache_beam.io.gcp.gcsio: INFO: Finished listing 30 files in 0.057604074478149414 seconds.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-runPerformanceTest-df-py37.xml
----------------------------------------------------------------------
XML: <https://ci-beam.apache.org/job/beam_PerformanceTests_WordCountIT_Py37/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 1 test in 820.482s
FAILED (errors=1)
> Task :sdks:python:test-suites:dataflow:py37:runPerformanceTest FAILED
:sdks:python:test-suites:dataflow:py37:runPerformanceTest (Thread[Execution **** for ':',5,main]) completed. Took 13 mins 43.268 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py37:runPerformanceTest'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 15m 0s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/wi2rgjir3h4ce
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org