You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/23 03:30:55 UTC
[GitHub] [airflow] juneoh opened a new issue #12556: Add ORC support to GCSToBigQueryOperator
juneoh opened a new issue #12556:
URL: https://github.com/apache/airflow/issues/12556
**Description**
[BigQuery API for loading data](https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobConfigurationLoad) lists ORC file format as one of the supported formats, but it is currently not allowed in GCSToBigQueryOperator. Specifying ORC as the source format raises a [ValueError in BigQueryHook](https://github.com/apache/airflow/blob/a3dfd04ce4224e80c7b32351a744980e72bc10aa/airflow/providers/google/cloud/hooks/bigquery.py#L1667).
**Use case / motivation**
Include ORC file format in BigQueryHook so that GCSToBigQueryOperator will fully allow all file formats supported by the BigQuery API.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek edited a comment on issue #12556: Add ORC support to GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
turbaszek edited a comment on issue #12556:
URL: https://github.com/apache/airflow/issues/12556#issuecomment-732020880
As mentioned in PR the `run_load` method is deprecated and users should use `insert_job` method which supports everything possible. That said, the change we should do is to update `GCSToBigQueryOperator` to use new method instead of the old one, see https://github.com/apache/airflow/issues/10288 (the PR seems to be in there but needs some more attention)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] juneoh commented on issue #12556: Add ORC support to GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
juneoh commented on issue #12556:
URL: https://github.com/apache/airflow/issues/12556#issuecomment-755939924
I've learned that holiday productivity is an illusion. I'll check in again next week as I'm ramping back up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] juneoh commented on issue #12556: Add ORC support to GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
juneoh commented on issue #12556:
URL: https://github.com/apache/airflow/issues/12556#issuecomment-767960816
Still working on this. The basic implementation is done but it's mostly a copy-paste from the BigQuery hook, so I'm trying to see what additional refactoring and testing is possible with the leverage of BigQuery Python SDK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #12556: Add ORC support to GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #12556:
URL: https://github.com/apache/airflow/issues/12556#issuecomment-732020880
As mentioned in PR the `run_load` method is deprecated and users should use `insert_job` method which supports everything possible. That said, the change we should do is to update `GCSToBigQueryOperator` to use new method instead of the old one, see https://github.com/apache/airflow/issues/10288
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #12556: Add ORC support to GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12556:
URL: https://github.com/apache/airflow/issues/12556#issuecomment-922950332
@juneoh are you still working on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] juneoh commented on issue #12556: Add ORC support to GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
juneoh commented on issue #12556:
URL: https://github.com/apache/airflow/issues/12556#issuecomment-760752685
Started working on this. Upon studying the history, looks like we need to fix the both paths in execute() for BigQuery and external tables to stop using the deprecated methods, which will likely to fix #10288 and #12329 along the way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #12556: Add ORC support to GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12556:
URL: https://github.com/apache/airflow/issues/12556#issuecomment-731908618
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #12556: Add ORC support to GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #12556:
URL: https://github.com/apache/airflow/issues/12556#issuecomment-749290219
@juneoh Are you willing to submit a PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] juneoh commented on issue #12556: Add ORC support to GCSToBigQueryOperator
Posted by GitBox <gi...@apache.org>.
juneoh commented on issue #12556:
URL: https://github.com/apache/airflow/issues/12556#issuecomment-749291510
Sure, I'll take a look during the holidays.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org