You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/30 13:27:59 UTC
[GitHub] [airflow] exploy opened a new issue #8645: BigQuery: integer range partitioning support for BQ tables
exploy opened a new issue #8645:
URL: https://github.com/apache/airflow/issues/8645
**Description**
Support for integer range partitioning for BigQuery tables
**Use case / motivation**
There is already support for time partitioning while creating BQ tables and Google offers more options than only time partitioning.
We found integer range partitioning working best for our case so we customized few classes to gain this functionality.
More details: https://cloud.google.com/bigquery/docs/partitioned-tables
Will contribute the change but it is not yet ready
**Related Issues**
A quick search revealed no similar issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #8645: BigQuery: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #8645:
URL: https://github.com/apache/airflow/issues/8645
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] totogo commented on issue #8645: BigQuery: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
totogo commented on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-748038904
@mik-laj I was looking for this feature too, it's not there yet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #8645: BigQuery: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-747137092
@exploy Could you please check if this feature is still missing? We have made major changes to this integration and many of the missing features are now available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] exploy commented on issue #8645: BigQuery: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
exploy commented on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-622075183
Related PR https://github.com/apache/airflow/pull/8653
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #8645: BigQuery: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-748085004
I was able to create tables with integer range partitioning.
```
create_table = BigQueryCreateEmptyTableOperator(
task_id="create_table",
dataset_id="test_dataset",
table_id="test_airflow_table",
table_resource={
"rangePartitioning": {
"field": "my_int_field",
"range": {
"start": 1,
"end": 10000,
"interval": 100
}
},
"id": "new-range-partition-table",
"schema": {
"fields": [
{
"name": "my_int_field",
"type": "INTEGER",
"mode": "NULLABLE"
}
]
}
}
)
````


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #8645: BigQuery: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-748085004
I was able to create tables with integer range partitioning.
```python
create_table = BigQueryCreateEmptyTableOperator(
task_id="create_table",
dataset_id="test_dataset",
table_id="test_airflow_table",
table_resource={
"rangePartitioning": {
"field": "my_int_field",
"range": {
"start": 1,
"end": 10000,
"interval": 100
}
},
"id": "new-range-partition-table",
"schema": {
"fields": [
{
"name": "my_int_field",
"type": "INTEGER",
"mode": "NULLABLE"
}
]
}
}
)
````


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #8645: BigQuery: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-621836578
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org