You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/18 18:35:24 UTC
[GitHub] [airflow] vchiapaikeo opened a new pull request, #28444: Fix GCSToBigQueryOperator not respecting schema_obj
vchiapaikeo opened a new pull request, #28444:
URL: https://github.com/apache/airflow/pull/28444
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #28441
related: #28441
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
GCSToBigQueryOperator allows multiple ways to specify schema of the BigQuery table:
1. Setting autodetect == True
1. Setting schema_fields directly with autodetect == False
1. Setting a schema_object and optionally a schema_object_bucket with autodetect == False
This third method seems to be broken in the latest provider version (8.6.0) and will always result in this error:
```
[2022-12-16, 21:06:18 UTC] {taskinstance.py:1772} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py", line 395, in execute
self.configuration = self._check_schema_fields(self.configuration)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py", line 524, in _check_schema_fields
raise RuntimeError(
RuntimeError: Table schema was not found. Set autodetect=True to automatically set schema fields from source objects or pass schema_fields explicitly
```
The reason for this is because [this block](https://github.com/apache/airflow/blob/25bdbc8e6768712bad6043618242eec9c6632618/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py#L318-L320) where `if self.schema_object and self.source_format != "DATASTORE_BACKUP":`. fails to set self.schema_fields. It only sets the local variable, schema_fields. When self._check_schema_fields is subsequently called [here](https://github.com/apache/airflow/blob/25bdbc8e6768712bad6043618242eec9c6632618/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py#L395), we enter the [first block](https://github.com/apache/airflow/blob/25bdbc8e6768712bad6043618242eec9c6632618/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py#L523-L528) because autodetect is false and schema_fields is not set.
This PR sets the instance variable, self.schema_fields when the user passes in a schema_obj. Additionally, it uses self.schema_object_bucket instead of the erroneous self.bucket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vchiapaikeo commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
vchiapaikeo commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1359965360
My pleasure @eladkal! Sure I can try to take a look at it this weekend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] axelborja commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
axelborja commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1385156952
Any idea in which release this one will be shipped ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1356850980
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vchiapaikeo commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
vchiapaikeo commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1358403044
cc @turbaszek, I believe you are codeowner. Can you approve this as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vchiapaikeo commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
vchiapaikeo commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1357794397
Whoops @eladkal - fixed the isort failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1358888647
Thanks @vchiapaikeo !
If you have time I'd appreciate also helping to resolve https://github.com/apache/airflow/issues/12329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal merged pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
eladkal merged PR #28444:
URL: https://github.com/apache/airflow/pull/28444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1405760068
> Hi, when will this be released? It is not in apache-airflow[gcp]==2.5.1.
Why do you think so ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #28444:
URL: https://github.com/apache/airflow/pull/28444#discussion_r1051961618
##########
airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:
##########
@@ -320,8 +320,10 @@ def execute(self, context: Context):
impersonation_chain=self.impersonation_chain,
)
if self.schema_object and self.source_format != "DATASTORE_BACKUP":
- schema_fields = json.loads(gcs_hook.download(self.bucket, self.schema_object).decode("utf-8"))
- self.log.info("Autodetected fields from schema object: %s", schema_fields)
+ self.schema_fields = json.loads(
+ gcs_hook.download(self.schema_object_bucket, self.schema_object).decode("utf-8")
+ )
+ self.log.info("Autodetected fields from schema object: %s", self.schema_fields)
Review Comment:
Correct me if I'm wrong but the entire block lines of 316-326 can be replaced with same code as we have in
https://github.com/apache/airflow/blob/199359bb3886699904ca075de7bd5fdfe5105c5f/airflow/providers/google/cloud/operators/bigquery.py#L1603-L1611
which is shorter and with no nesting if
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] grimwm commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by "grimwm (via GitHub)" <gi...@apache.org>.
grimwm commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1416602449
> > Hi, when will this be released? It is not in apache-airflow[gcp]==2.5.1.
>
> Why do you think so ?
Hi, that was my fault. I didn't realize our production Airflow was on 2.5.0. I finally put it together and got it sorted and forgot to update here. Sorry, and thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1358889857
Awesome work, congrats on your first merged pull request!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vchiapaikeo commented on a diff in pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
vchiapaikeo commented on code in PR #28444:
URL: https://github.com/apache/airflow/pull/28444#discussion_r1052195894
##########
airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:
##########
@@ -320,8 +320,10 @@ def execute(self, context: Context):
impersonation_chain=self.impersonation_chain,
)
if self.schema_object and self.source_format != "DATASTORE_BACKUP":
- schema_fields = json.loads(gcs_hook.download(self.bucket, self.schema_object).decode("utf-8"))
- self.log.info("Autodetected fields from schema object: %s", schema_fields)
+ self.schema_fields = json.loads(
+ gcs_hook.download(self.schema_object_bucket, self.schema_object).decode("utf-8")
+ )
+ self.log.info("Autodetected fields from schema object: %s", self.schema_fields)
Review Comment:
Yes, good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1385293654
<img width="1001" alt="Screenshot 2023-01-17 at 12 35 50" src="https://user-images.githubusercontent.com/595491/212889145-0c8232a5-7a7d-45a7-ba60-d54919194c67.png">
Seen the above ? ^^ looks like this is already shipped in the previous provider and you can update the provider when you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] grimwm commented on pull request #28444: Fix GCSToBigQueryOperator not respecting schema_obj
Posted by "grimwm (via GitHub)" <gi...@apache.org>.
grimwm commented on PR #28444:
URL: https://github.com/apache/airflow/pull/28444#issuecomment-1405298383
Hi, when will this be released? It is not in 2.5.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org