You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/12 03:43:24 UTC
[GitHub] [airflow] mbettan opened a new issue #13626: Airflow Operator "GoogleCloudStorageToBigQueryOperator" - BQ data load through External Source Partitioning [Hive Partitioning]
mbettan opened a new issue #13626:
URL: https://github.com/apache/airflow/issues/13626
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
Composer Airflow Operator "GoogleCloudStorageToBigQueryOperator" needs the functionality to support BQ data load through External Source Partitioning [Hive Partitioning]
**Use case / motivation**
Ability to ingest data from data sources with years historical data into Google BigQuery for analytics purposes - we would like to support data load through External Source Partitioning [Hive Partitioning]
**Are you willing to submit a PR?**
N/A
**Related Issues**
BQ Load Bash Operator is the workaround but introduced other issues with the Airflow connection ID usage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13626: Airflow Operator "GoogleCloudStorageToBigQueryOperator" - BQ data load through External Source Partitioning [Hive Partitioning]
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13626:
URL: https://github.com/apache/airflow/issues/13626#issuecomment-760133980
This looks like an invalid ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mbettan commented on issue #13626: Airflow Operator "GoogleCloudStorageToBigQueryOperator" - BQ data load through External Source Partitioning [Hive Partitioning]
Posted by GitBox <gi...@apache.org>.
mbettan commented on issue #13626:
URL: https://github.com/apache/airflow/issues/13626#issuecomment-760259992
Thank you @mik-laj - this solved my issue we used as alternative: BigQueryInsertJobOperator
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13626: Airflow Operator "GoogleCloudStorageToBigQueryOperator" - BQ data load through External Source Partitioning [Hive Partitioning]
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13626:
URL: https://github.com/apache/airflow/issues/13626#issuecomment-758375566
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] smowden edited a comment on issue #13626: Airflow Operator "GoogleCloudStorageToBigQueryOperator" - BQ data load through External Source Partitioning [Hive Partitioning]
Posted by GitBox <gi...@apache.org>.
smowden edited a comment on issue #13626:
URL: https://github.com/apache/airflow/issues/13626#issuecomment-758580541
check out the docs here:
https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobConfigurationLoad
if you insert a load job (see #13598) you should be able to provide hivePartitioningOptions:
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#HivePartitioningOptions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13626: Airflow Operator "GoogleCloudStorageToBigQueryOperator" - BQ data load through External Source Partitioning [Hive Partitioning]
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13626:
URL: https://github.com/apache/airflow/issues/13626#issuecomment-758505230
Have you tried to use BigQueryInsertJobOperator? for example, see: https://github.com/apache/airflow/pull/13598
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #13626: Airflow Operator "GoogleCloudStorageToBigQueryOperator" - BQ data load through External Source Partitioning [Hive Partitioning]
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #13626:
URL: https://github.com/apache/airflow/issues/13626
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] smowden commented on issue #13626: Airflow Operator "GoogleCloudStorageToBigQueryOperator" - BQ data load through External Source Partitioning [Hive Partitioning]
Posted by GitBox <gi...@apache.org>.
smowden commented on issue #13626:
URL: https://github.com/apache/airflow/issues/13626#issuecomment-758580541
check out the docs here:
https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobConfigurationLoad
if you insert a load job (see #13598) you should be able to prive hivePartitioningOptions:
https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#HivePartitioningOptions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org