You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/11 07:46:32 UTC
[GitHub] [airflow] chenmic opened a new issue #10288: gcs_to_bigquery using deprecated methods
chenmic opened a new issue #10288:
URL: https://github.com/apache/airflow/issues/10288
Hello,
I'm running Airflow 1.10.10 and using the `GCSToBigQueryOperator` from `airflow.providers.google.cloud.transfers.gcs_to_bigquery` in order to load data from google cloud storage into a big query table.
Although everything is working as expected, I keep seeing (a lot of) deprecation warnings in my logs:
`DeprecationWarning: The bigquery_conn_id parameter has been deprecated. You should pass the gcp_conn_id parameter.`
`bigquery.py:103: DeprecationWarning: This method will be deprecated. Please use `BigQueryHook.get_client` method`
`taskinstance.py:984: DeprecationWarning: This method is deprecated. Please use `airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_load``
`bigquery.py:1614: DeprecationWarning: This method is deprecated. Please use `BigQueryHook.insert_job` method.`
Although everything is working well, I'm afraid that in the future those warnings will indeed cause problems.
Please update the code and find a way to use the suggested methods instead of the deprecated ones.
If any more information is needed from me regarding the warnings or my system I will be happy to add it.
Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aprettyloner commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
aprettyloner commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-906012109
My team is also seeing the deprecation warning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aprettyloner commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
aprettyloner commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-906558936
> @aprettyloner are you willing to submit a PR to fix it?
Absolutely. Pretty tied in up migrating our DAGs from 1.10 to 2.1 (plus training new hires) but I am happy to open this once things settle down. I'll check on this issue status to ensure not duplicating work when the time comes. Thanks @eladkal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-671993341
@coopergillan I assigned you to this ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-671860452
Thanks for the issue @chenmic ! I think same goes for other BQ transfer operators as they were not refactored to use new methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy edited a comment on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
kazanzhy edited a comment on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-988323907
Hi all. @chenmic @aprettyloner
I've tried to remove warnings during GCSToBigQueryOperator as in PR #20119
But as a result, I copied almost all code from `bq_hook.create_external_table` and `bq_hook.run_load` which made the operator's execute method very sophisticated.
That's why I just switched from `cursor` method calls to `BigQueryHook` methods. In my opinion, these methods should not be deprecated.
We need someplace for complex dict creation, like `configuration` for `bq_hook.insert_job()` or `table_recource` for `bq_hook.create_empty_table()`.
In my opinion, better to keep this code in BigQueryHook and don't move to Operator. But maybe I'm wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] coopergillan commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
coopergillan commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-671990741
@mik-laj - I'm happy to take it as a P4. Plenty else to work on, but I can grab it when time allows.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aprettyloner edited a comment on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
aprettyloner edited a comment on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-906012109
My team is also seeing the deprecation warning.
```
/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:31 DeprecationWarning: This decorator is deprecated.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aprettyloner edited a comment on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
aprettyloner edited a comment on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-906012109
My team is also seeing the deprecation warning for decorator in `gcs_to_bigquery` on all `GCSToBigQueryOperator` imports. This is noisy and leads some of our developers to think we need to make a code change.
```
/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:31 DeprecationWarning: This decorator is deprecated.
In previous versions, all subclasses of BaseOperator must use apply_default decorator for the`default_args` feature to work properly.
In current version, it is optional. The decorator is applied automatically using the metaclass.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-671786853
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] coopergillan commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
coopergillan commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-671976103
Is this one anything of a priority?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aprettyloner edited a comment on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
aprettyloner edited a comment on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-906012109
My team is also seeing the deprecation warning for decorator in `gcs_to_bigquery` on all `GCSToBigQueryOperator` imports. This is noisy as it is not coming from our own code.
```
/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:31 DeprecationWarning: This decorator is deprecated.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-906168461
@aprettyloner are you willing to submit a PR to fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aprettyloner edited a comment on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
aprettyloner edited a comment on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-906012109
My team is also seeing the deprecation warning for decorator in `gcs_to_bigquery` on all `GCSToBigQueryOperator` imports. This is noisy and is misleading some of our developers to think we need to make a code change.
```
/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:31 DeprecationWarning: This decorator is deprecated.
In previous versions, all subclasses of BaseOperator must use apply_default decorator for the`default_args` feature to work properly.
In current version, it is optional. The decorator is applied automatically using the metaclass.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #10288:
URL: https://github.com/apache/airflow/issues/10288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-671982564
This is a P4 level ticket. However, it is worth fixing this problem.
```
"P1" means Critical Impact – Service Unusable in Production
"P2" means High Impact – Service Use Severely Impaired
"P3" means Medium Impact – Service Use Partially Impaired
"P4" means Low Impact – Service Fully Usable
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-982789776
@aprettyloner ping :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kazanzhy commented on issue #10288: gcs_to_bigquery using deprecated methods
Posted by GitBox <gi...@apache.org>.
kazanzhy commented on issue #10288:
URL: https://github.com/apache/airflow/issues/10288#issuecomment-988323907
Hi all.
I've tried to remove warnings during GCSToBigQueryOperator as in PR #20119
But as a result, I copied almost all code from `bq_hook.create_external_table` and `bq_hook.run_load` which made the operator's execute method very sophisticated.
That's why I just switched from `cursor` method calls to `BigQueryHook` methods. In my opinion, these methods should not be deprecated.
We need someplace for complex dict creation, like `configuration` for `bq_hook.insert_job()` or `table_recource` for `bq_hook.create_empty_table()`.
In my opinion, better to keep this code in BigQueryHook and don't move to Operator. But maybe I'm wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org