You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/14 08:20:08 UTC
[GitHub] [airflow] VladaZakharova opened a new pull request, #27052: Add deferrable mode to GCPToBigQueryOperator + tests
VladaZakharova opened a new pull request, #27052:
URL: https://github.com/apache/airflow/pull/27052
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] quentin-sommer commented on pull request #27052: Add deferrable mode to GCPToBigQueryOperator + tests
Posted by GitBox <gi...@apache.org>.
quentin-sommer commented on PR #27052:
URL: https://github.com/apache/airflow/pull/27052#issuecomment-1341679823
Hi, I tracked an issue I have to this pr:
```
def _check_schema_fields(self, table_resource):
```
(https://github.com/apache/airflow/pull/27052/files#diff-875bf3d1bfbba7067dc754732c0e416b8ebe7a5b722bc9ac428b98934f04a16fR512)
is executed and try to infer a schema on my newline delimited json file. It obviously doesn't work, I end up with column names containing `{"`.
I'm trying to understand why the function is called on all the data types without checking if `source_format` is `'CSV'`. Maybe I'm just using it wrong? Hope you can help
I could provide the `schema_fields` every time to make the operator bypass the check but it was convenient to let bigquery detect the schema.
ping @VladaZakharova and @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VladaZakharova commented on pull request #27052: Add deferrable mode to GCPToBigQueryOperator + tests
Posted by GitBox <gi...@apache.org>.
VladaZakharova commented on PR #27052:
URL: https://github.com/apache/airflow/pull/27052#issuecomment-1344013789
@quentin-sommer
Hi Team,
I am working on preparing a PR with a fix for this issue and will share all the updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #27052: Add deferrable mode to GCPToBigQueryOperator + tests
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #27052:
URL: https://github.com/apache/airflow/pull/27052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27052: Add deferrable mode to GCPToBigQueryOperator + tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27052:
URL: https://github.com/apache/airflow/pull/27052#issuecomment-1288198810
Needs conflict resolving after string normalization in providers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VladaZakharova commented on pull request #27052: Add deferrable mode to GCPToBigQueryOperator + tests
Posted by GitBox <gi...@apache.org>.
VladaZakharova commented on PR #27052:
URL: https://github.com/apache/airflow/pull/27052#issuecomment-1345366891
@quentin-sommer
Hi Team :)
Please, check the PR that I have created for the fix to this issue:
https://github.com/apache/airflow/pull/28284
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org