You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@beam.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2020/04/01 18:35:50 UTC
Build failed in Jenkins: beam_PostCommit_Python35 #2139
See <https://builds.apache.org/job/beam_PostCommit_Python35/2139/display/redirect?page=changes>
Changes:
[zyichi] Update BigQuery source in bigquery_tornadoes example
[rohde.samuel] Add dependency comment in streaming cache
[ehudm] [BEAM-1894] Remove obsolete EagerRunner test
------------------------------------------
[...truncated 8.01 MB...]
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:47.830Z: JOB_MESSAGE_BASIC: Finished operation read from datastore/Reshuffle/ReshufflePerKey/GroupByKey/Read+read from datastore/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+read from datastore/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+read from datastore/Reshuffle/RemoveRandomKeys+read from datastore/Read+Globally/CombineGlobally(CountCombineFn)/KeyWithVoid+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Reify+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:47.905Z: JOB_MESSAGE_BASIC: Executing operation Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:47.965Z: JOB_MESSAGE_BASIC: Finished operation Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.051Z: JOB_MESSAGE_BASIC: Executing operation Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Globally/CombineGlobally(CountCombineFn)/UnKey
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.347Z: JOB_MESSAGE_BASIC: Finished operation read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+row to string+count/CombineGlobally(CountCombineFn)/KeyWithVoid+count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Reify+count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.432Z: JOB_MESSAGE_DEBUG: Value "read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough).cleanup_signal" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.475Z: JOB_MESSAGE_BASIC: Executing operation count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.519Z: JOB_MESSAGE_BASIC: Executing operation read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.545Z: JOB_MESSAGE_BASIC: Finished operation count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.565Z: JOB_MESSAGE_BASIC: Finished operation read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.631Z: JOB_MESSAGE_BASIC: Executing operation count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+count/CombineGlobally(CountCombineFn)/UnKey
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.664Z: JOB_MESSAGE_DEBUG: Value "read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.793Z: JOB_MESSAGE_BASIC: Executing operation read/_PassThroughThenCleanup/Create/Read+read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/ParDo(RemoveJsonFiles)
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:59.689Z: JOB_MESSAGE_BASIC: Finished operation Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Globally/CombineGlobally(CountCombineFn)/UnKey
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:59.806Z: JOB_MESSAGE_DEBUG: Value "Globally/CombineGlobally(CountCombineFn)/UnKey.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:59.966Z: JOB_MESSAGE_BASIC: Executing operation Globally/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:00.008Z: JOB_MESSAGE_BASIC: Finished operation Globally/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:00.069Z: JOB_MESSAGE_DEBUG: Value "Globally/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:00.129Z: JOB_MESSAGE_BASIC: Executing operation Globally/CombineGlobally(CountCombineFn)/DoOnce/Read+Globally/CombineGlobally(CountCombineFn)/InjectDefault/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.754Z: JOB_MESSAGE_BASIC: Finished operation Globally/CombineGlobally(CountCombineFn)/DoOnce/Read+Globally/CombineGlobally(CountCombineFn)/InjectDefault/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.842Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:01.806Z: JOB_MESSAGE_BASIC: Finished operation read/_PassThroughThenCleanup/Create/Read+read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/ParDo(RemoveJsonFiles)
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.851Z: JOB_MESSAGE_BASIC: Finished operation count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+count/CombineGlobally(CountCombineFn)/UnKey
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.960Z: JOB_MESSAGE_DEBUG: Value "count/CombineGlobally(CountCombineFn)/UnKey.out" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:06.046Z: JOB_MESSAGE_BASIC: Executing operation count/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:06.121Z: JOB_MESSAGE_BASIC: Finished operation count/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:06.208Z: JOB_MESSAGE_DEBUG: Value "count/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:06.353Z: JOB_MESSAGE_BASIC: Executing operation count/CombineGlobally(CountCombineFn)/DoOnce/Read+count/CombineGlobally(CountCombineFn)/InjectDefault/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.897Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.977Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Read+assert_that/Group/GroupByKey/GroupByWindow+assert_that/Group/Map(_merge_tagged_vals_under_key)+assert_that/Unkey+assert_that/Match
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:15.162Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Read+assert_that/Group/GroupByKey/GroupByWindow+assert_that/Group/Map(_merge_tagged_vals_under_key)+assert_that/Unkey+assert_that/Match
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:15.243Z: JOB_MESSAGE_DEBUG: Executing success step success49
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:15.377Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:15.442Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:15.471Z: JOB_MESSAGE_BASIC: Stopping worker pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:12.061Z: JOB_MESSAGE_BASIC: Finished operation count/CombineGlobally(CountCombineFn)/DoOnce/Read+count/CombineGlobally(CountCombineFn)/InjectDefault/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:12.176Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:12.258Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:12.349Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Read+assert_that/Group/GroupByKey/GroupByWindow+assert_that/Group/Map(_merge_tagged_vals_under_key)+assert_that/Unkey+assert_that/Match
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:21.514Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Read+assert_that/Group/GroupByKey/GroupByWindow+assert_that/Group/Map(_merge_tagged_vals_under_key)+assert_that/Unkey+assert_that/Match
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:21.619Z: JOB_MESSAGE_DEBUG: Executing success step success40
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:22.304Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:22.352Z: JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:22.387Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:21.775Z: JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:21.874Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:21.918Z: JOB_MESSAGE_BASIC: Stopping worker pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-04-01_11_24_57-11894884384059897893 is in state JOB_STATE_DONE
INFO:apache_beam.io.gcp.tests.bigquery_matcher:Attempting to perform query SELECT name, language FROM python_bq_file_loads_1585765477684.output_table1 to BQ
DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254
DEBUG:google.auth.transport._http_client:Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal:80
DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 192
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bigquery.googleapis.com:443
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/queries/3ebc1b47-a534-418d-b338-71f32978e101?maxResults=0&location=US HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anonfe808ac11b24b22de19473db82f1fcded616be8e/data HTTP/1.1" 200 None
INFO:apache_beam.io.gcp.tests.bigquery_matcher:Result of query is: [('flink', 'java'), ('flink', 'scala'), ('spark', 'scala'), ('spark', 'py'), ('spark', 'scala'), ('beam', 'py'), ('beam', 'java'), ('beam', 'go')]
INFO:apache_beam.io.gcp.tests.bigquery_matcher:Attempting to perform query SELECT name, foundation FROM python_bq_file_loads_1585765477684.output_table2 to BQ
DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254
DEBUG:google.auth.transport._http_client:Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal:80
DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 192
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bigquery.googleapis.com:443
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/queries/db638bb8-5aaa-4f01-9458-fb04bde37e43?maxResults=0&location=US HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anon1b43990a7ebf311ab68df50df170d9add0d97d6f/data HTTP/1.1" 200 None
INFO:apache_beam.io.gcp.tests.bigquery_matcher:Result of query is: [('flink', 'apache'), ('spark', 'apache'), ('beam', 'apache')]
INFO:apache_beam.io.gcp.tests.bigquery_matcher:Attempting to perform query SELECT name, language FROM python_bq_file_loads_1585765477684.output_table3 to BQ
DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254
DEBUG:google.auth.transport._http_client:Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal:80
DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 192
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bigquery.googleapis.com:443
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/queries/1c48bd0a-e153-4c30-b9da-1eff82e8b4aa?maxResults=0&location=US HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anon76d123eec7f074c32c066e6c140bbcfec8c20386/data HTTP/1.1" 200 None
INFO:apache_beam.io.gcp.tests.bigquery_matcher:Result of query is: [('beam', 'go'), ('beam', 'py'), ('spark', 'py'), ('beam', 'java'), ('flink', 'java'), ('flink', 'scala'), ('spark', 'scala'), ('spark', 'scala')]
INFO:apache_beam.io.gcp.tests.bigquery_matcher:Attempting to perform query SELECT name, foundation FROM python_bq_file_loads_1585765477684.output_table4 to BQ
DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254
DEBUG:google.auth.transport._http_client:Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal:80
DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 192
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bigquery.googleapis.com:443
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/queries/43cc7280-6966-4be9-9363-373020682ea7?maxResults=0&location=US HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anon979d83537faa78c1aa3c158f35540b91c806ce04/data HTTP/1.1" 200 None
INFO:apache_beam.io.gcp.tests.bigquery_matcher:Result of query is: [('beam', 'apache'), ('flink', 'apache'), ('spark', 'apache')]
INFO:apache_beam.io.gcp.bigquery_file_loads_test:Deleting dataset python_bq_file_loads_1585765477684 in project apache-beam-testing
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:36.430Z: JOB_MESSAGE_DETAILED: Checking permissions granted to controller Service Account.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:37.954Z: JOB_MESSAGE_BASIC: Finished operation read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.022Z: JOB_MESSAGE_DEBUG: Value "read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough).cleanup_signal" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.100Z: JOB_MESSAGE_BASIC: Executing operation read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.163Z: JOB_MESSAGE_BASIC: Finished operation read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0)
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.242Z: JOB_MESSAGE_DEBUG: Value "read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0).output" materialized.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.327Z: JOB_MESSAGE_BASIC: Executing operation read/_PassThroughThenCleanup/Create/Read+read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/ParDo(RemoveJsonFiles)
WARNING:apache_beam.options.pipeline_options:--region not set; will default to us-central1. Future releases of Beam will require the user to set --region explicitly, or else have a default set via the gcloud tool. https://cloud.google.com/compute/docs/regions-zones
INFO:apache_beam.io.gcp.bigquery_file_loads_test:Created dataset python_bq_file_loads_1585766138451 in project apache-beam-testing
DEBUG:apache_beam.io.gcp.bigquery_tools:Created the table with id output_table1
INFO:apache_beam.io.gcp.bigquery_tools:Created table apache-beam-testing.python_bq_file_loads_1585766138451.output_table1 with schema <TableSchema
fields: [<TableFieldSchema
fields: []
mode: 'NULLABLE'
name: 'name'
type: 'STRING'>, <TableFieldSchema
fields: []
mode: 'NULLABLE'
name: 'language'
type: 'STRING'>]>. Result: <Table
creationTime: 1585766139493
etag: 'm+zyxtJvlxUouzbvg8jJTw=='
id: 'apache-beam-testing:python_bq_file_loads_1585766138451.output_table1'
kind: 'bigquery#table'
lastModifiedTime: 1585766139638
location: 'US'
numBytes: 0
numLongTermBytes: 0
numRows: 0
schema: <TableSchema
fields: [<TableFieldSchema
fields: []
mode: 'NULLABLE'
name: 'name'
type: 'STRING'>, <TableFieldSchema
fields: []
mode: 'NULLABLE'
name: 'language'
type: 'STRING'>]>
selfLink: 'https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_bq_file_loads_1585766138451/tables/output_table1'
tableReference: <TableReference
datasetId: 'python_bq_file_loads_1585766138451'
projectId: 'apache-beam-testing'
tableId: 'output_table1'>
type: 'TABLE'>.
DEBUG:apache_beam.io.gcp.bigquery_tools:Created the table with id output_table2
INFO:apache_beam.io.gcp.bigquery_tools:Created table apache-beam-testing.python_bq_file_loads_1585766138451.output_table2 with schema <TableSchema
fields: [<TableFieldSchema
fields: []
mode: 'NULLABLE'
name: 'name'
type: 'STRING'>, <TableFieldSchema
fields: []
mode: 'NULLABLE'
name: 'foundation'
type: 'STRING'>]>. Result: <Table
creationTime: 1585766139899
etag: 'svQ8MWgUNzEInmMgCQ/e/w=='
id: 'apache-beam-testing:python_bq_file_loads_1585766138451.output_table2'
kind: 'bigquery#table'
lastModifiedTime: 1585766140038
location: 'US'
numBytes: 0
numLongTermBytes: 0
numRows: 0
schema: <TableSchema
fields: [<TableFieldSchema
fields: []
mode: 'NULLABLE'
name: 'name'
type: 'STRING'>, <TableFieldSchema
fields: []
mode: 'NULLABLE'
name: 'foundation'
type: 'STRING'>]>
selfLink: 'https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_bq_file_loads_1585766138451/tables/output_table2'
tableReference: <TableReference
datasetId: 'python_bq_file_loads_1585766138451'
projectId: 'apache-beam-testing'
tableId: 'output_table2'>
type: 'TABLE'>.
WARNING:apache_beam.options.pipeline_options:--region not set; will default to us-central1. Future releases of Beam will require the user to set --region explicitly, or else have a default set via the gcloud tool. https://cloud.google.com/compute/docs/regions-zones
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.277Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.308Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35/src/sdks/python/apache_beam/io/gcp/bigquery.py:1416: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:863: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
temp_location = p.options.view_as(GoogleCloudOptions).temp_location
WARNING:apache_beam.runners.dataflow.dataflow_runner:Typical end users should not use this worker jar feature. It can only be used when FnAPI is enabled.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/pipeline.pb...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/pipeline.pb in 0 seconds.
INFO:apache_beam.runners.portability.stager:Executing command: ['/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35/src/build/gradleenv/-1734967054/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', 'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:40.611Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Create/Read+assert_that/Group/pair_with_0+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:40.701Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:40.758Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Close
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:40.840Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Read+assert_that/Group/GroupByKey/GroupByWindow+assert_that/Group/Map(_merge_tagged_vals_under_key)+assert_that/Unkey+assert_that/Match
INFO:apache_beam.runners.portability.stager:Copying Beam SDK "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35/src/sdks/python/build/apache-beam.tar.gz" to staging location.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/requirements.txt...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/requirements.txt in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/six-1.14.0.tar.gz...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/six-1.14.0.tar.gz in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/parameterized-0.7.1.tar.gz...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/parameterized-0.7.1.tar.gz in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/mock-2.0.0.tar.gz...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/mock-2.0.0.tar.gz in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/pbr-5.4.4.tar.gz...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/pbr-5.4.4.tar.gz in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/funcsigs-1.0.2.tar.gz...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/funcsigs-1.0.2.tar.gz in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/PyHamcrest-1.10.1.tar.gz...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/PyHamcrest-1.10.1.tar.gz in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/dataflow_python_sdk.tar...
INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/dataflow_python_sdk.tar in 0 seconds.
INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/dataflow-worker.jar...
FATAL: command execution failed
hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from 165.171.154.104.bc.googleusercontent.com/104.154.171.165:40434 failed. The channel is closing down or has closed down
at hudson.remoting.Channel.call(Channel.java:950)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
at com.sun.proxy.$Proxy140.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1150)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1142)
at hudson.Launcher$ProcStarter.join(Launcher.java:470)
at hudson.plugins.gradle.Gradle.perform(Gradle.java:317)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: java.nio.channels.ClosedChannelException
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142)
at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
ERROR: apache-beam-jenkins-12 is offline; cannot locate JDK 1.8 (latest)
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Jenkins build is back to normal : beam_PostCommit_Python35 #2142
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python35/2142/display/redirect?page=changes>
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python35 #2141
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python35/2141/display/redirect?page=changes>
Changes:
[daniel.o.programmer] [BEAM-9642] Create runtime invokers for SDF methods.
------------------------------------------
[...truncated 10.53 MB...]
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>: response: <{'x-xss-protection': '0', 'status': '403', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'date': 'Thu, 02 Apr 2020 00:02:16 GMT', 'server': 'ESF', 'content-type': 'application/json; charset=UTF-8', 'x-frame-options': 'SAMEORIGIN', '-content-encoding': 'gzip', 'vary': 'Origin, X-Origin, Referer', 'cache-control': 'private', 'content-length': '478'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:02:21.863Z: JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 218, in execute
self._split_task)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 226, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 263, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 669, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 728, in _export_files
include_header=False)
File "/usr/local/lib/python3.5/site-packages/apache_beam/utils/retry.py", line 236, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 715, in perform_extract_job
response = self.client.jobs.Insert(request)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>: response: <{'x-xss-protection': '0', 'status': '403', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'date': 'Thu, 02 Apr 2020 00:02:19 GMT', 'server': 'ESF', 'content-type': 'application/json; charset=UTF-8', 'x-frame-options': 'SAMEORIGIN', '-content-encoding': 'gzip', 'vary': 'Origin, X-Origin, Referer', 'cache-control': 'private', 'content-length': '478'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:02:25.174Z: JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 218, in execute
self._split_task)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 226, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 263, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 669, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 728, in _export_files
include_header=False)
File "/usr/local/lib/python3.5/site-packages/apache_beam/utils/retry.py", line 236, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 715, in perform_extract_job
response = self.client.jobs.Insert(request)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>: response: <{'x-xss-protection': '0', 'status': '403', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'date': 'Thu, 02 Apr 2020 00:02:23 GMT', 'server': 'ESF', 'content-type': 'application/json; charset=UTF-8', 'x-frame-options': 'SAMEORIGIN', '-content-encoding': 'gzip', 'vary': 'Origin, X-Origin, Referer', 'cache-control': 'private', 'content-length': '478'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:02:28.459Z: JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 218, in execute
self._split_task)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 226, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 263, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 669, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 728, in _export_files
include_header=False)
File "/usr/local/lib/python3.5/site-packages/apache_beam/utils/retry.py", line 236, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 715, in perform_extract_job
response = self.client.jobs.Insert(request)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>: response: <{'x-xss-protection': '0', 'status': '403', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'date': 'Thu, 02 Apr 2020 00:02:26 GMT', 'server': 'ESF', 'content-type': 'application/json; charset=UTF-8', 'x-frame-options': 'SAMEORIGIN', '-content-encoding': 'gzip', 'vary': 'Origin, X-Origin, Referer', 'cache-control': 'private', 'content-length': '478'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:02:28.745Z: JOB_MESSAGE_BASIC: Finished operation read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+months with tornadoes+monthly count/GroupByKey+monthly count/Combine/Partial+monthly count/GroupByKey/Reify+monthly count/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:02:28.839Z: JOB_MESSAGE_DEBUG: Executing failure step failure22
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:02:28.872Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: S02:read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+months with tornadoes+monthly count/GroupByKey+monthly count/Combine/Partial+monthly count/GroupByKey/Reify+monthly count/GroupByKey/Write failed., Internal Issue (b351eea7d6070b87): 63963027:24514
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:02:29.164Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:02:29.228Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:02:29.262Z: JOB_MESSAGE_BASIC: Stopping worker pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:04:19.831Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:04:19.875Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-02T00:04:19.911Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-04-01_16_56_35-12587617880379681448 is in state JOB_STATE_FAILED
apache_beam.io.gcp.tests.utils: INFO: Clean up a BigQuery table with project: apache-beam-testing, dataset: BigQueryTornadoesIT, table: monthly_tornadoes_1585785380334.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 192
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "DELETE /bigquery/v2/projects/apache-beam-testing/datasets/BigQueryTornadoesIT/tables/monthly_tornadoes_1585785380334 HTTP/1.1" 404 None
--------------------- >> end captured logging << ---------------------
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_16_56_41-17566071964017717399?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_11_27-11279165765814616715?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_21_10-14392231141847606159?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_30_21-11322941583118322641?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_39_00-17350961585757899088?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_47_00-16878921466663899064?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_16_56_36-7388031278101078375?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_19_07-11533606860101021534?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_28_19-13132068702668230056?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_36_59-12827731749070390360?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_45_04-17142741413330043314?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_16_56_39-4792182057925141920?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_08_51-15980711773453333993?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_17_57-11997375112431958673?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_26_17-16503755008421809944?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_34_51-8801543965688203621?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_43_34-13959563133429753594?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_16_56_35-10635133450793389666?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_17_56-2464712606423761047?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_26_50-8805569670159955346?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_35_08-5155753613889245737?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_44_31-8182932995282448065?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_16_56_35-12969164060353860917?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_05_32-6555924885351485887?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_13_36-8792004516234363760?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_22_26-4375643573973660516?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_31_27-5278044983685492070?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_39_09-985121282021911920?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_48_24-7383678958614858346?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_16_56_35-12587617880379681448?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_04_44-15864447585057044500?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_13_57-2336846713817614170?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_22_41-15030866574824407999?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_32_09-1336952709735786262?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_39_38-1788274055435707148?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_46_55-379332211185402571?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_55_30-14192125012182383030?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_16_56_36-4081543926284201631?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_06_34-8302738415550199168?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_20_00-9882069299951277730?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_28_36-13840888884691130792?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_46_17-18011007151274378001?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_16_56_37-6651187893124460233?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_05_15-12073855395586663458?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_15_00-8701128034989314838?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_23_33-7852132439126389354?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_32_43-15608643038356799023?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_41_22-15876770311456162164?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_17_50_06-2016210348051958170?project=apache-beam-testing
----------------------------------------------------------------------
XML: nosetests-postCommitIT-df-py35.xml
----------------------------------------------------------------------
XML: <https://builds.apache.org/job/beam_PostCommit_Python35/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 59 tests in 4068.779s
FAILED (SKIP=9, errors=2)
> Task :sdks:python:test-suites:dataflow:py35:postCommitIT FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python35/ws/src/sdks/python/test-suites/dataflow/py35/build.gradle'> line: 56
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py35:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 9m 2s
86 actionable tasks: 64 executed, 22 from cache
Publishing build scan...
https://gradle.com/s/i73a7ggz26zd2
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org
Build failed in Jenkins: beam_PostCommit_Python35 #2140
Posted by Apache Jenkins Server <je...@builds.apache.org>.
See <https://builds.apache.org/job/beam_PostCommit_Python35/2140/display/redirect?page=changes>
Changes:
[robertwb] [BEAM-9340] Populate requirement for timer families.
------------------------------------------
[...truncated 10.51 MB...]
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>: response: <{'x-content-type-options': 'nosniff', 'vary': 'Origin, X-Origin, Referer', 'cache-control': 'private', '-content-encoding': 'gzip', 'content-type': 'application/json; charset=UTF-8', 'date': 'Wed, 01 Apr 2020 21:48:31 GMT', 'content-length': '478', 'server': 'ESF', 'status': '403', 'x-frame-options': 'SAMEORIGIN', 'transfer-encoding': 'chunked', 'x-xss-protection': '0'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:48:37.090Z: JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 218, in execute
self._split_task)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 226, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 263, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 669, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 728, in _export_files
include_header=False)
File "/usr/local/lib/python3.5/site-packages/apache_beam/utils/retry.py", line 236, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 715, in perform_extract_job
response = self.client.jobs.Insert(request)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>: response: <{'x-content-type-options': 'nosniff', 'vary': 'Origin, X-Origin, Referer', 'cache-control': 'private', '-content-encoding': 'gzip', 'content-type': 'application/json; charset=UTF-8', 'date': 'Wed, 01 Apr 2020 21:48:35 GMT', 'content-length': '478', 'server': 'ESF', 'status': '403', 'x-frame-options': 'SAMEORIGIN', 'transfer-encoding': 'chunked', 'x-xss-protection': '0'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:48:40.369Z: JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 218, in execute
self._split_task)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 226, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 263, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 669, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 728, in _export_files
include_header=False)
File "/usr/local/lib/python3.5/site-packages/apache_beam/utils/retry.py", line 236, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 715, in perform_extract_job
response = self.client.jobs.Insert(request)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>: response: <{'x-content-type-options': 'nosniff', 'vary': 'Origin, X-Origin, Referer', 'cache-control': 'private', '-content-encoding': 'gzip', 'content-type': 'application/json; charset=UTF-8', 'date': 'Wed, 01 Apr 2020 21:48:38 GMT', 'content-length': '478', 'server': 'ESF', 'status': '403', 'x-frame-options': 'SAMEORIGIN', 'transfer-encoding': 'chunked', 'x-xss-protection': '0'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:48:43.659Z: JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/batchworker.py", line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 218, in execute
self._split_task)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 226, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.5/site-packages/dataflow_worker/executor.py", line 263, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 669, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery.py", line 728, in _export_files
include_header=False)
File "/usr/local/lib/python3.5/site-packages/apache_beam/utils/retry.py", line 236, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py", line 715, in perform_extract_job
response = self.client.jobs.Insert(request)
File "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>: response: <{'x-content-type-options': 'nosniff', 'vary': 'Origin, X-Origin, Referer', 'cache-control': 'private', '-content-encoding': 'gzip', 'content-type': 'application/json; charset=UTF-8', 'date': 'Wed, 01 Apr 2020 21:48:41 GMT', 'content-length': '478', 'server': 'ESF', 'status': '403', 'x-frame-options': 'SAMEORIGIN', 'transfer-encoding': 'chunked', 'x-xss-protection': '0'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:48:43.910Z: JOB_MESSAGE_BASIC: Finished operation read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+months with tornadoes+monthly count/GroupByKey+monthly count/Combine/Partial+monthly count/GroupByKey/Reify+monthly count/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:48:44.032Z: JOB_MESSAGE_DEBUG: Executing failure step failure22
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:48:44.068Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: S02:read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+months with tornadoes+monthly count/GroupByKey+monthly count/Combine/Partial+monthly count/GroupByKey/Reify+monthly count/GroupByKey/Write failed., Internal Issue (7e755db2dd34273b): 63963027:24514
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:48:44.213Z: JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:48:44.324Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:48:44.359Z: JOB_MESSAGE_BASIC: Stopping worker pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:50:25.397Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:50:25.572Z: JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T21:50:25.671Z: JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job 2020-04-01_14_41_55-332000418083384536 is in state JOB_STATE_FAILED
apache_beam.io.gcp.tests.utils: INFO: Clean up a BigQuery table with project: apache-beam-testing, dataset: BigQueryTornadoesIT, table: monthly_tornadoes_1585777294237.
google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 192
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "DELETE /bigquery/v2/projects/apache-beam-testing/datasets/BigQueryTornadoesIT/tables/monthly_tornadoes_1585777294237 HTTP/1.1" 404 None
--------------------- >> end captured logging << ---------------------
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_41_55-6497161370811613398?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_01_48-9508792997637332249?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_10_21-18137221900557943530?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_18_59-14792124653882278193?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_27_28-14634152210450482759?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_41_59-8885308050103388199?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_56_30-8834493737500648963?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_07_07-6205130664694029407?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_15_48-3938053716478250919?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_24_36-245818642624159918?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_33_58-14539415178223242066?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_41_57-16705380871449534425?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_54_14-11278760919187506273?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_02_15-16899928904927479438?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_11_48-7226818908564208385?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_19_41-159156361073104918?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_28_32-10373978938788907892?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_41_56-915844550753565813?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_50_39-12597500532897823421?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_00_35-3121737760817522916?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_09_28-14779775917645183529?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_18_34-9798942446839447867?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_27_03-9191692635931387363?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_35_15-13491764075140189144?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_41_56-5222428469106404748?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_01_38-8355911117482645523?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_09_52-12227256901516614779?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_18_14-1322975552117912006?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_25_29-6904430435267383368?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_33_00-18397865603011103998?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_41_08-9164858917762264691?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_41_55-332000418083384536?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_50_54-12214712358840976316?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_58_42-4499767159122796260?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_07_44-13235316986496074907?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_16_25-4171569624150677464?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_24_42-8884166709475707527?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_33_42-6069506200283529231?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_41_57-10277912576432285335?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_51_55-11412869701663937129?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_03_30-3279403676903423219?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_11_46-17097499538820339151?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_29_14-17574021455961645307?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_41_58-2670800242498093136?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_14_50_51-4807990862994942336?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_00_31-9383865811394431829?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_09_10-6106453882463787946?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_18_51-14646006685781125898?project=apache-beam-testing
Worker logs: https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_15_27_59-1004025078609057622?project=apache-beam-testing
----------------------------------------------------------------------
XML: nosetests-postCommitIT-df-py35.xml
----------------------------------------------------------------------
XML: <https://builds.apache.org/job/beam_PostCommit_Python35/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 59 tests in 4096.447s
FAILED (SKIP=9, errors=2)
> Task :sdks:python:test-suites:dataflow:py35:postCommitIT FAILED
FAILURE: Build failed with an exception.
* Where:
Build file '<https://builds.apache.org/job/beam_PostCommit_Python35/ws/src/sdks/python/test-suites/dataflow/py35/build.gradle'> line: 56
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py35:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 10m 52s
86 actionable tasks: 67 executed, 19 from cache
Publishing build scan...
https://gradle.com/s/7xfhxe4sugwba
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: builds-unsubscribe@beam.apache.org
For additional commands, e-mail: builds-help@beam.apache.org