You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Valentyn Tymofieiev (Jira)" <ji...@apache.org> on 2020/05/01 16:23:00 UTC
[jira] [Assigned] (BEAM-9148) test flakiness:
BigQueryQueryToTableIT.test_big_query_standard_sql
[ https://issues.apache.org/jira/browse/BEAM-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Valentyn Tymofieiev reassigned BEAM-9148:
-----------------------------------------
Assignee: Pablo Estrada (was: Boyuan Zhang)
> test flakiness: BigQueryQueryToTableIT.test_big_query_standard_sql
> ------------------------------------------------------------------
>
> Key: BEAM-9148
> URL: https://issues.apache.org/jira/browse/BEAM-9148
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp, sdk-py-core, test-failures
> Reporter: Udi Meiri
> Assignee: Pablo Estrada
> Priority: Major
>
> There might be other flaky test cases from the same class, but I'm focusing on test_big_query_standard_sql here.
> {code}
> 19:39:12 ======================================================================
> 19:39:12 FAIL: test_big_query_standard_sql (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 19:39:12 ----------------------------------------------------------------------
> 19:39:12 Traceback (most recent call last):
> 19:39:12 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py", line 172, in test_big_query_standard_sql
> 19:39:12 big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 19:39:12 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py", line 84, in run_bq_pipeline
> 19:39:12 result = p.run()
> 19:39:12 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/testing/test_pipeline.py", line 112, in run
> 19:39:12 else test_runner_api))
> 19:39:12 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py", line 461, in run
> 19:39:12 self._options).run(False)
> 19:39:12 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py", line 474, in run
> 19:39:12 return self.runner.run_pipeline(self, self._options)
> 19:39:12 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py", line 53, in run_pipeline
> 19:39:12 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 19:39:12 AssertionError:
> 19:39:12 Expected: (Test pipeline expected terminated in state: DONE and Expected checksum is 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72)
> 19:39:12 but: Expected checksum is 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72 Actual checksum is da39a3ee5e6b4b0d3255bfef95601890afd80709
> 19:39:12
> 19:39:12 -------------------- >> begin captured logging << --------------------
> 19:39:12 root: DEBUG: Unhandled type_constraint: Union[]
> 19:39:12 root: DEBUG: Unhandled type_constraint: Union[]
> 19:39:12 apache_beam.runners.direct.direct_runner: INFO: Running pipeline with DirectRunner.
> 19:39:12 apache_beam.io.gcp.bigquery_tools: DEBUG: Query SELECT * FROM (SELECT "apple" as fruit) UNION ALL (SELECT "orange" as fruit) does not reference any tables.
> 19:39:12 apache_beam.io.gcp.bigquery_tools: WARNING: Dataset apache-beam-testing:temp_dataset_90f5797bdb5f4137af750399f91a8e66 does not exist so we will create it as temporary with location=None
> 19:39:12 apache_beam.io.gcp.bigquery: DEBUG: Creating or getting table <TableReference
> 19:39:12 datasetId: 'python_query_to_table_15792323245106'
> 19:39:12 projectId: 'apache-beam-testing'
> 19:39:12 tableId: 'output_table'> with schema {'fields': [{'name': 'fruit', 'type': 'STRING', 'mode': 'NULLABLE'}]}.
> 19:39:12 apache_beam.io.gcp.bigquery_tools: DEBUG: Created the table with id output_table
> 19:39:12 apache_beam.io.gcp.bigquery_tools: INFO: Created table apache-beam-testing.python_query_to_table_15792323245106.output_table with schema <TableSchema
> 19:39:12 fields: [<TableFieldSchema
> 19:39:12 fields: []
> 19:39:12 mode: 'NULLABLE'
> 19:39:12 name: 'fruit'
> 19:39:12 type: 'STRING'>]>. Result: <Table
> 19:39:12 creationTime: 1579232328576
> 19:39:12 etag: 'WYysl6UIvc8IWMmTiiKhbg=='
> 19:39:12 id: 'apache-beam-testing:python_query_to_table_15792323245106.output_table'
> 19:39:12 kind: 'bigquery#table'
> 19:39:12 lastModifiedTime: 1579232328629
> 19:39:12 location: 'US'
> 19:39:12 numBytes: 0
> 19:39:12 numLongTermBytes: 0
> 19:39:12 numRows: 0
> 19:39:12 schema: <TableSchema
> 19:39:12 fields: [<TableFieldSchema
> 19:39:12 fields: []
> 19:39:12 mode: 'NULLABLE'
> 19:39:12 name: 'fruit'
> 19:39:12 type: 'STRING'>]>
> 19:39:12 selfLink: 'https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_query_to_table_15792323245106/tables/output_table'
> 19:39:12 tableReference: <TableReference
> 19:39:12 datasetId: 'python_query_to_table_15792323245106'
> 19:39:12 projectId: 'apache-beam-testing'
> 19:39:12 tableId: 'output_table'>
> 19:39:12 type: 'TABLE'>.
> 19:39:12 apache_beam.io.gcp.bigquery: DEBUG: Attempting to flush to all destinations. Total buffered: 2
> 19:39:12 apache_beam.io.gcp.bigquery: DEBUG: Flushing data to apache-beam-testing:python_query_to_table_15792323245106.output_table. Total 2 rows.
> 19:39:12 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Attempting to perform query SELECT fruit from `python_query_to_table_15792323245106.output_table`; to BQ
> 19:39:12 google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
> 19:39:12 google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
> 19:39:12 urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
> 19:39:12 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
> 19:39:12 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
> 19:39:12 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
> 19:39:12 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
> 19:39:12 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 181
> 19:39:12 urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): www.googleapis.com:443
> 19:39:12 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None
> 19:39:12 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/queries/032dabb2-92e3-472b-993e-f0e840ba0104?maxResults=0&location=US HTTP/1.1" 200 None
> 19:39:12 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anon2425bb619e2b0611267244ec96ed403c7aae8d57/data HTTP/1.1" 200 None
> 19:39:12 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Read from given query (SELECT fruit from `python_query_to_table_15792323245106.output_table`;), total rows 0
> 19:39:12 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Generate checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709
> 19:39:12 --------------------- >> end captured logging << ---------------------
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1387/timestamps/?time=HH:mm:ss&timeZone=GMT-8&appendLog&locale=en_US
> {code}
> 04:09:22 ======================================================================
> 04:09:22 FAIL: test_big_query_standard_sql (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 04:09:22 ----------------------------------------------------------------------
> 04:09:22 Traceback (most recent call last):
> 04:09:22 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py", line 172, in test_big_query_standard_sql
> 04:09:22 big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 04:09:22 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py", line 84, in run_bq_pipeline
> 04:09:22 result = p.run()
> 04:09:22 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/testing/test_pipeline.py", line 113, in run
> 04:09:22 else test_runner_api))
> 04:09:22 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py", line 461, in run
> 04:09:22 self._options).run(False)
> 04:09:22 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py", line 474, in run
> 04:09:22 return self.runner.run_pipeline(self, self._options)
> 04:09:22 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py", line 53, in run_pipeline
> 04:09:22 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 04:09:22 AssertionError:
> 04:09:22 Expected: (Test pipeline expected terminated in state: DONE and Expected checksum is 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72)
> 04:09:22 but: Expected checksum is 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72 Actual checksum is da39a3ee5e6b4b0d3255bfef95601890afd80709
> 04:09:22
> 04:09:22 -------------------- >> begin captured logging << --------------------
> 04:09:22 root: DEBUG: Unhandled type_constraint: Union[]
> 04:09:22 root: DEBUG: Unhandled type_constraint: Union[]
> 04:09:22 apache_beam.runners.direct.direct_runner: INFO: Running pipeline with DirectRunner.
> 04:09:22 apache_beam.io.gcp.bigquery_tools: DEBUG: Query SELECT * FROM (SELECT "apple" as fruit) UNION ALL (SELECT "orange" as fruit) does not reference any tables.
> 04:09:22 apache_beam.io.gcp.bigquery_tools: WARNING: Dataset apache-beam-testing:temp_dataset_25b82ddd70424425ac110ac6a6de303c does not exist so we will create it as temporary with location=None
> 04:09:22 apache_beam.io.gcp.bigquery: DEBUG: Creating or getting table <TableReference
> 04:09:22 datasetId: 'python_query_to_table_15791765509862'
> 04:09:22 projectId: 'apache-beam-testing'
> 04:09:22 tableId: 'output_table'> with schema {'fields': [{'name': 'fruit', 'type': 'STRING', 'mode': 'NULLABLE'}]}.
> 04:09:22 apache_beam.io.gcp.bigquery_tools: DEBUG: Created the table with id output_table
> 04:09:22 apache_beam.io.gcp.bigquery_tools: INFO: Created table apache-beam-testing.python_query_to_table_15791765509862.output_table with schema <TableSchema
> 04:09:22 fields: [<TableFieldSchema
> 04:09:22 fields: []
> 04:09:22 mode: 'NULLABLE'
> 04:09:22 name: 'fruit'
> 04:09:22 type: 'STRING'>]>. Result: <Table
> 04:09:22 creationTime: 1579176555984
> 04:09:22 etag: 'qPxrZxyNtuoTRqGhw9zi3Q=='
> 04:09:22 id: 'apache-beam-testing:python_query_to_table_15791765509862.output_table'
> 04:09:22 kind: 'bigquery#table'
> 04:09:22 lastModifiedTime: 1579176556029
> 04:09:22 location: 'US'
> 04:09:22 numBytes: 0
> 04:09:22 numLongTermBytes: 0
> 04:09:22 numRows: 0
> 04:09:22 schema: <TableSchema
> 04:09:22 fields: [<TableFieldSchema
> 04:09:22 fields: []
> 04:09:22 mode: 'NULLABLE'
> 04:09:22 name: 'fruit'
> 04:09:22 type: 'STRING'>]>
> 04:09:22 selfLink: 'https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_query_to_table_15791765509862/tables/output_table'
> 04:09:22 tableReference: <TableReference
> 04:09:22 datasetId: 'python_query_to_table_15791765509862'
> 04:09:22 projectId: 'apache-beam-testing'
> 04:09:22 tableId: 'output_table'>
> 04:09:22 type: 'TABLE'>.
> 04:09:22 apache_beam.io.gcp.bigquery: DEBUG: Attempting to flush to all destinations. Total buffered: 2
> 04:09:22 apache_beam.io.gcp.bigquery: DEBUG: Flushing data to apache-beam-testing:python_query_to_table_15791765509862.output_table. Total 2 rows.
> 04:09:22 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Attempting to perform query SELECT fruit from `python_query_to_table_15791765509862.output_table`; to BQ
> 04:09:22 google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254
> 04:09:22 google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id
> 04:09:22 urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
> 04:09:22 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
> 04:09:22 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80
> 04:09:22 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144
> 04:09:22 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token
> 04:09:22 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 181
> 04:09:22 urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): www.googleapis.com:443
> 04:09:22 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None
> 04:09:22 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/queries/1c114200-2ef8-4916-9084-ba6e38cc9e4a?maxResults=0&location=US HTTP/1.1" 200 None
> 04:09:22 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anon046b15f69112e6f2ad0b7c87ec9f1684fd8ab3bf/data HTTP/1.1" 200 None
> 04:09:22 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Read from given query (SELECT fruit from `python_query_to_table_15791765509862.output_table`;), total rows 0
> 04:09:22 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Generate checksum: da39a3ee5e6b4b0d3255bfef95601890afd80709
> 04:09:22 --------------------- >> end captured logging << ---------------------
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1379/timestamps/?time=HH:mm:ss&timeZone=GMT-8&appendLog&locale=en_US
> (note there are 2 test failures in this second log)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)