You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "FaresDev8 (via GitHub)" <gi...@apache.org> on 2025/07/10 09:15:05 UTC
[I] Null key reference in bigquery cursor query results [airflow]
FaresDev8 opened a new issue, #53127:
URL: https://github.com/apache/airflow/issues/53127
### Apache Airflow Provider(s)
google
### Versions of Apache Airflow Providers
Issue faced on two versions:
**apache-airflow-providers-google==12.0.0
apache-airflow-providers-google==15.1.0**
### Apache Airflow version
2.10.5
### Operating System
Debian Linux 12
### Deployment
Docker-Compose
### Deployment details
Docker version 27.5.1
No tools used other than airflow.
### What happened
Code reference that causes the issue:
https://github.com/apache/airflow/blob/e142ab96a0ecbb953e9fb8ab6d17b4c2a7624aba/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py#L1663-L1666
https://github.com/apache/airflow/blob/e142ab96a0ecbb953e9fb8ab6d17b4c2a7624aba/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py#L2061-L2062
While executing an external table creation job, I faced an issue that the above code runs into an error:
```
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/google/cloud/hooks/bigquery.py", line 2077, in _format_schema_for_description
for field in schema["fields"]:
~~~~~~^^^^^^^^^^
KeyError: 'fields'
```
After debugging I noticed that this kind of jobs return an empty dictionary for the key "schema"
This causes the second code reference in line 2061-2062 to try to loop over an empty dictionary and then it fails as there are no key "fields".
### What you think should happen instead
What should happen is that the following script should check if the value for key "schema" exists and not check if the key it self exists
https://github.com/apache/airflow/blob/e142ab96a0ecbb953e9fb8ab6d17b4c2a7624aba/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py#L1663-L1666
If we make the logic to check the value instead of the key then it will make sure that it won't call function '**_format_schema_for_description()**' without a proper schema values.
To fix this we can modify the script at line 1663-1666 to the following script:
`if query_results["schema"]:`
` self.description = _format_schema_for_description(query_results["schema"])`
`else:`
` self.description = []`
### How to reproduce
To reproduce the problem you need the following:
1- Airflow and airflow-providers installed < versions dont matter much as this script exists in multiple versions.
2- Create a sql script that create an external table < This will assure no schema will be returned.
3- execute the sql script using the cursor from the bigqueryhook
`hook = BigQueryHook(gcp_conn_id=gcp_conn_id, use_legacy_sql=False)
conn = hook.get_conn()
cursor = conn.cursor()
cursor.execute(operation='SELECT 1 AS NUMBER')`
### Anything else
This problem is part of the script and will appear whenever anyone tries to execute a non resulting query.
Something like creating and dropping tables, data transformation, and other system queries.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Null key reference in bigquery cursor query results [airflow]
Posted by "FaresDev8 (via GitHub)" <gi...@apache.org>.
FaresDev8 commented on issue #53127:
URL: https://github.com/apache/airflow/issues/53127#issuecomment-3066717376
@jroachgolf84 you are correct, even if we check for schema values that will not resolve the issue.
Will look into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Null key reference in bigquery cursor query results [airflow]
Posted by "prohackerzzz (via GitHub)" <gi...@apache.org>.
prohackerzzz commented on issue #53127:
URL: https://github.com/apache/airflow/issues/53127#issuecomment-3066758032
what if we just use this it will not cause any exception if there are no fields
schema = query_results.get("schema", {})
fields = schema.get("fields", [])
self.description = _format_schema_for_description(fields) if fields else []
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Null key reference in bigquery cursor query results [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #53127:
URL: https://github.com/apache/airflow/issues/53127#issuecomment-3056490781
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Null key reference in bigquery cursor query results [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #53127:
URL: https://github.com/apache/airflow/issues/53127#issuecomment-3057209993
I'm not quite sure if this code snippet below will do the trick. Even if `query_results["schema"]` exists, that does not guarantee that the `"fields"` key will exist. The change needs to be made in here: https://github.com/apache/airflow/blob/e142ab96a0ecbb953e9fb8ab6d17b4c2a7624aba/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py#L2061-L2062
```python
# This will NOT fix the issue
if query_results["schema"]:
self.description = _format_schema_for_description(query_results["schema"])
else:
self.description = []
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Null key reference in bigquery cursor query results [airflow]
Posted by "FaresDev8 (via GitHub)" <gi...@apache.org>.
FaresDev8 commented on issue #53127:
URL: https://github.com/apache/airflow/issues/53127#issuecomment-3066949638
Hey @prohackerzzz, as @jroachgolf84 mentioned the solution need to be in the (_format_schema_for_description) function it self here:
https://github.com/apache/airflow/blob/e142ab96a0ecbb953e9fb8ab6d17b4c2a7624aba/providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py#L2061-L2062
The solution can be as simple as adding an if statement to check if the "fields" key exists and since this function only loops on "fields" then this check would do.
`if "fields" in schema:`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Null key reference in bigquery cursor query results [airflow]
Posted by "prohackerzzz (via GitHub)" <gi...@apache.org>.
prohackerzzz commented on issue #53127:
URL: https://github.com/apache/airflow/issues/53127#issuecomment-3066724558
Hi! I'd like to work on this issue. Please let me know if that's okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Null key reference in bigquery cursor query results [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #53127: Null key reference in bigquery cursor query results
URL: https://github.com/apache/airflow/issues/53127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org