[GitHub] [airflow] patricker opened a new issue, #26785: BigQueryHook Requires Optional Field When Parsing Results Schema

[GitBox <gi...@apache.org>]

patricker opened a new issue, #26785:
URL: https://github.com/apache/airflow/issues/26785

   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-google 8.3.0
   
   ### Apache Airflow version
   
   2.3.4
   
   ### Operating System
   
   OSX
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   When querying BigQuery using the BigQueryHook, sometimes the following error is returned:
   
   ```
   [2022-09-29, 19:04:57 UTC] {{taskinstance.py:1902}} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 171, in execute
       return_value = self.execute_callable()
     File "/usr/local/lib/python3.9/site-packages/airflow/operators/python.py", line 189, in execute_callable
       return self.python_callable(*self.op_args, **self.op_kwargs)
     File "/usr/src/app/dags/test.py", line 23, in curried_bigquery
       cursor.execute(UPDATE_HIGHWATER_MARK_QUERY)
     File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2700, in execute
       self.description = _format_schema_for_description(query_results["schema"])
     File "/usr/local/lib/python3.9/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 3009, in _format_schema_for_description
       field["mode"] == "NULLABLE",
   KeyError: 'mode'
   ```
   
   
   ### What you think should happen instead
   
   The schema should be returned without error.
   
   ### How to reproduce
   
   _No response_
   
   ### Anything else
   
   According to the official docs, only `name` and `type` are required to be present. `mode` is listed as optional.
   https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableFieldSchema
   
   The code currently expects `mode` to be present.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #26785: BigQueryHook Requires Optional Field When Parsing Results Schema

[GitBox <gi...@apache.org>]

potiuk closed issue #26785: BigQueryHook Requires Optional Field When Parsing Results Schema
URL: https://github.com/apache/airflow/issues/26785


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org