You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "X33834 (via GitHub)" <gi...@apache.org> on 2026/09/13 17:22:04 UTC
[PR] fix(google): Support Table objects in BigQueryHook.create_table table_resource [airflow]
X33834 opened a new pull request, #73099:
URL: https://github.com/apache/airflow/pull/73099
### Problem
Closes: #73091
`BigQueryHook.create_table()`'s `table_resource` parameter is typed and documented to accept a `dict`, `Table`, `TableReference`, or `TableListItem`, but the implementation only worked with a plain `dict`:
- `Table` instance: `Table.from_api_repr()` was called backwards (it expects an API-JSON dict, not a `Table` object) → `TypeError: argument of type 'Table' is not iterable`
- `TableReference` / `TableListItem`: raw `**`-unpacking on a non-mapping → `TypeError: 'TableReference' object is not a mapping`
This is reachable from `BigQueryCreateTableOperator`, which has the identical type hint and forwards `table_resource` straight into the hook. A DAG author who follows the documented type hint and passes a `Table` obtained from `get_table()` / `list_tables()` hits a crash.
### Solution
Convert any object input to a plain dict via `.to_api_repr()` before merging `schema_fields` and building the final `Table`. A bare `TableReference` is wrapped under `"tableReference"` since its `to_api_repr()` is flat, unlike `Table` / `TableListItem`.
### Tests
Added unit tests in `TestTableOperations` covering all four input forms (`dict`, `Table`, `TableReference`, `TableListItem`) plus schema merging:
- `test_create_table_with_table_object`
- `test_create_table_with_table_reference_object`
- `test_create_table_with_table_list_item`
- `test_create_table_with_table_object_and_schema_fields`
Ran locally: conversion logic verified against real `google.cloud.bigquery` objects; `ruff check` and `py_compile` clean. Full provider test suite runs in CI via breeze.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "X33834 (via GitHub)" <gi...@apache.org>.
X33834 commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5950020043
Hi @molcay, thanks a lot for running the BigQuery system tests on my branch — really appreciate it!
Quick note: the `Static checks` failure comes from the breeze doc-image regeneration hooks (`dev/breeze/doc/images/output_workflow-run*`), unrelated to the actual PR changes. I've updated the branch with the latest `main` to clear it.
Heads-up: the CI workflows on the new commit are currently in `action_required` state — would you mind re-approving/triggering them when you get a chance? Looking forward to the system test results. 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #73099:
URL: https://github.com/apache/airflow/pull/73099
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #73099:
URL: https://github.com/apache/airflow/pull/73099#discussion_r4172067101
##########
newsfragments/73099.feature.rst:
##########
@@ -0,0 +1 @@
+Support ``Table`` objects in ``BigQueryHook.create_table`` for ``table_resource``
Review Comment:
Please remove the newsfragment, we don't use it for providers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "X33834 (via GitHub)" <gi...@apache.org>.
X33834 commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5883155585
Friendly ping — both review points were addressed on Sep 17 (generative-AI disclosure added to the PR description, conventional-commit prefix removed from the title). GitHub checks are green.
Happy to adjust anything else — could someone take another look when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5999820808
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "X33834 (via GitHub)" <gi...@apache.org>.
X33834 commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5718036810
Thanks @shahar1 for the review. Both points addressed:
1. Added the generative-AI disclosure section to the PR description per the PR template.
2. Removed the conventional-commit prefix from the title.
Ready for another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "X33834 (via GitHub)" <gi...@apache.org>.
X33834 commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5968772298
Both pushed to the branch:
- Added a `create_table_from_table_object` task to `example_bigquery_tables.py` passing a `google.cloud.bigquery.Table` instance (same schema as the dict example), wired into the chain right after `create_table`, so both supported input types of `BigQueryHook.create_table` stay covered.
- Removed the newsfragment.
CI is running; happy to add a `TableListItem` example too if you think it's worth covering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "molcay (via GitHub)" <gi...@apache.org>.
molcay commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5885871506
Hi @X33834,
Thank you for your PR! The change seems GDM.
In the google provider side, we want to also see the system tests results after a change.
Did you also run the system tests which are related to the BigQuery service?
If so, please share your results as well.
> Since you change the `BigQueryHook`, it is good to run all BigQuery related system tests
Thank you in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(google): Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5654841241
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "Morningstar202604 (via GitHub)" <gi...@apache.org>.
Morningstar202604 commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5946719567
Thanks for the pointer, @molcay. I don't have access to a GCP project with the BigQuery API enabled, so I can't run the system tests from here. If someone on the team can trigger `providers/google/tests/system/google/cloud/bigquery/` on this branch, I'd be glad to look at any failures.
To make the change easy to reason about while those run:
- `create_table` now normalizes `Table` / `TableReference` / `TableListItem` inputs through `to_api_repr()` before calling the API; the dict path is unchanged, so existing system tests that pass dicts exercise the same code path as before.
- The BigQuery unit suite in this PR's CI is green, including the new cases covering each accepted input type.
Happy to adjust the implementation if the system tests surface anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "X33834 (via GitHub)" <gi...@apache.org>.
X33834 commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5886551752
Thanks @molcay!
The PR's CI run includes the BigQuery unit test suite and is green (`providers/google/tests/unit/google/cloud/hooks/test_bigquery.py`, including the new cases for `Table`/`TableReference`/`TableListItem` inputs).
For the system tests under `providers/google/tests/system/google/cloud/bigquery/`: I don't have access to a GCP project with the BigQuery API enabled, so I couldn't run them myself. Happy to run them if credentials can be provided, or feel free to trigger them on your side if that's easier.
For what it's worth, the change is contained to input normalization in `create_table`: it accepts `Table`/`TableReference`/`TableListItem` objects and converts them via `to_api_repr()` (previously a `Table` object was passed through `Table.from_api_repr`, which raised). The dict path is unchanged, so existing system tests that pass dicts exercise the same code path as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "X33834 (via GitHub)" <gi...@apache.org>.
X33834 commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5946721715
Thanks for the pointer, @molcay. I don't have access to a GCP project with the BigQuery API enabled, so I can't run the system tests from here. If someone on the team can trigger `providers/google/tests/system/google/cloud/bigquery/` on this branch, I'd be glad to look at any failures.
To make the change easy to reason about while those run:
- `create_table` now normalizes `Table` / `TableReference` / `TableListItem` inputs through `to_api_repr()` before calling the API; the dict path is unchanged, so existing system tests that pass dicts exercise the same code path as before.
- The BigQuery unit suite in this PR's CI is green, including the new cases covering each accepted input type.
Happy to adjust the implementation if the system tests surface anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "molcay (via GitHub)" <gi...@apache.org>.
molcay commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5961874034
Hi @X33834,
It seems like your change did not break anything.
Since your change is about handling different kind of table definitions for the `BigQueryCreateTableOperator`; we must also add more tasks to the `bigquery_tables` system test.
As you can (see)[https://github.com/X33834/airflow/blob/fix-bigquery-create-table/providers/google/tests/system/google/cloud/bigquery/example_bigquery_tables.py#L85-L97], we already have one `create_table` task with dict config for `table_resource` argument. We must add the other type example in this DAG.
Can you make this change as well to make the system test future-proofed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Table objects in BigQueryHook.create_table table_resource [airflow]
Posted by "molcay (via GitHub)" <gi...@apache.org>.
molcay commented on PR #73099:
URL: https://github.com/apache/airflow/pull/73099#issuecomment-5948222353
Hi @X33834,
I am trying to run the system tests for bigquery on your branch, when I have a result I will let you know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org