You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/10/05 20:08:17 UTC
[GitHub] [beam] ahmedabu98 commented on a diff in pull request #23070: Support TIMESTAMP type in BigQueryIO with BEAM_ROW output type, and in read_gbq
ahmedabu98 commented on code in PR #23070:
URL: https://github.com/apache/beam/pull/23070#discussion_r986013514
##########
sdks/python/apache_beam/io/gcp/bigquery_schema_tools_test.py:
##########
@@ -61,6 +61,25 @@ def test_check_conversion_with_empty_schema(self):
the_table_schema=schema)
self.assertEqual(usertype.__annotations__, {})
+ def test_check_schema_conversions_with_timestamp(self):
+ fields = [
+ bigquery.TableFieldSchema(name='stn', type='STRING', mode="NULLABLE"),
+ bigquery.TableFieldSchema(name='temp', type='FLOAT64', mode="REPEATED"),
+ bigquery.TableFieldSchema(
+ name='times', type='TIMESTAMP', mode="NULLABLE")
+ ]
+ schema = bigquery.TableSchema(fields=fields)
+
+ usertype = bigquery_schema_tools.generate_user_type_from_bq_schema(
+ the_table_schema=schema)
+ self.assertEqual(
+ usertype.__annotations__,
+ {
+ 'stn': typing.Optional[str],
+ 'temp': typing.Sequence[np.float64],
+ 'times': typing.Any
Review Comment:
```
'times': typing.Any
```
Should this be testing for an Optional `datetime.datetime` type?
##########
sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py:
##########
@@ -258,6 +258,123 @@ def test_table_schema_retrieve_with_direct_read(self):
utype(id=4, name='customer2', type='test')
]))
+ @pytest.mark.it_postcommit
+ def test_table_schema_retrieve_with_timestamp(self):
+ the_table = bigquery_tools.BigQueryWrapper().get_table(
+ project_id="apache-beam-testing",
+ dataset_id="beam_bigquery_io_test",
+ table_id="taxi_small")
+ table = the_table.schema
+ utype = bigquery_schema_tools. \
+ generate_user_type_from_bq_schema(table)
+ with beam.Pipeline(argv=self.args) as p:
+ result = (
+ p | apache_beam.io.gcp.bigquery.ReadFromBigQuery(
+ gcs_location="gs://bqio_schema_test",
+ dataset="beam_bigquery_io_test",
+ table="taxi_small",
+ project="apache-beam-testing",
+ output_type='BEAM_ROW'))
+ assert_that(
+ result,
+ equal_to([
+ utype(
+ event_timestamp=datetime.datetime(
+ 2021,
+ 11,
+ 5,
+ 20,
+ 57,
+ 43,
+ 612000,
+ tzinfo=datetime.timezone.utc),
+ ride_id='0004b5de-8db1-425b-8eec-7a25b04a1ee0',
+ point_idx=9,
+ latitude=40.757380000000005,
+ timestamp='2021-11-05T16:57:43.60906-04:00',
+ meter_reading=0.37846154,
+ meter_increment=0.04205128,
+ ride_status='enroute',
+ passenger_count=1,
+ longitude=-73.98969000000001),
Review Comment:
Is it possible to only test for the timestamp field since that's what this test is focusing on? It would help make the test more concise and easy to read in the future.
One way is to change it so that ReadFromBigQuery queries for the timestamp field. If schema conversion isn't implemented yet for query reads, another way may be to add a step that filters `result` so you have a PCollection of timestamp elements.
##########
sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py:
##########
@@ -258,6 +258,123 @@ def test_table_schema_retrieve_with_direct_read(self):
utype(id=4, name='customer2', type='test')
]))
+ @pytest.mark.it_postcommit
+ def test_table_schema_retrieve_with_timestamp(self):
+ the_table = bigquery_tools.BigQueryWrapper().get_table(
+ project_id="apache-beam-testing",
+ dataset_id="beam_bigquery_io_test",
+ table_id="taxi_small")
+ table = the_table.schema
+ utype = bigquery_schema_tools. \
+ generate_user_type_from_bq_schema(table)
Review Comment:
```suggestion
schema = the_table.schema
utype = bigquery_schema_tools. \
generate_user_type_from_bq_schema(schema)
```
To avoid confusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org