You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/30 13:27:59 UTC

[GitHub] [airflow] exploy opened a new issue #8645: BigQuery: integer range partitioning support for BQ tables

exploy opened a new issue #8645:
URL: https://github.com/apache/airflow/issues/8645


   **Description**
   
   Support for  integer range partitioning for BigQuery tables
   
   **Use case / motivation**
   
   There is already support for time partitioning while creating BQ tables and Google offers more options than only time partitioning. 
   We found integer range partitioning working best for our case so we customized few classes to gain this functionality. 
   
   More details: https://cloud.google.com/bigquery/docs/partitioned-tables
   
   Will contribute the change but it is not yet ready
   
   **Related Issues**
   
   A quick search revealed no similar issues. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj closed issue #8645: BigQuery: integer range partitioning support for BQ tables

Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #8645:
URL: https://github.com/apache/airflow/issues/8645


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] totogo commented on issue #8645: BigQuery: integer range partitioning support for BQ tables

Posted by GitBox <gi...@apache.org>.
totogo commented on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-748038904


   @mik-laj I was looking for this feature too, it's not there yet


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on issue #8645: BigQuery: integer range partitioning support for BQ tables

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-747137092


   @exploy Could you please check if this feature is still missing? We have made major changes to this integration and many of the missing features are now available.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] exploy commented on issue #8645: BigQuery: integer range partitioning support for BQ tables

Posted by GitBox <gi...@apache.org>.
exploy commented on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-622075183


   Related PR https://github.com/apache/airflow/pull/8653


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on issue #8645: BigQuery: integer range partitioning support for BQ tables

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-748085004


   I was able to create tables with integer range partitioning.
   ```
       create_table = BigQueryCreateEmptyTableOperator(
           task_id="create_table",
           dataset_id="test_dataset",
           table_id="test_airflow_table",
           table_resource={
               "rangePartitioning": {
                   "field": "my_int_field",
                   "range": {
                       "start": 1,
                       "end": 10000,
                       "interval": 100
                   }
               },
               "id": "new-range-partition-table",
               "schema": {
                   "fields": [
                       {
                           "name": "my_int_field",
                           "type": "INTEGER",
                           "mode": "NULLABLE"
                       }
                   ]
               }
           }
       )
   ````
   ![Screenshot 2020-12-18 at 14 28 03](https://user-images.githubusercontent.com/12058428/102619916-7c59c300-413d-11eb-8f21-7fae8ea20b2f.png)
   ![Screenshot 2020-12-18 at 14 28 18](https://user-images.githubusercontent.com/12058428/102619923-7d8af000-413d-11eb-9434-afcb097b17b4.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj edited a comment on issue #8645: BigQuery: integer range partitioning support for BQ tables

Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-748085004


   I was able to create tables with integer range partitioning.
   ```python
       create_table = BigQueryCreateEmptyTableOperator(
           task_id="create_table",
           dataset_id="test_dataset",
           table_id="test_airflow_table",
           table_resource={
               "rangePartitioning": {
                   "field": "my_int_field",
                   "range": {
                       "start": 1,
                       "end": 10000,
                       "interval": 100
                   }
               },
               "id": "new-range-partition-table",
               "schema": {
                   "fields": [
                       {
                           "name": "my_int_field",
                           "type": "INTEGER",
                           "mode": "NULLABLE"
                       }
                   ]
               }
           }
       )
   ````
   ![Screenshot 2020-12-18 at 14 28 03](https://user-images.githubusercontent.com/12058428/102619916-7c59c300-413d-11eb-8f21-7fae8ea20b2f.png)
   ![Screenshot 2020-12-18 at 14 28 18](https://user-images.githubusercontent.com/12058428/102619923-7d8af000-413d-11eb-9434-afcb097b17b4.png)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #8645: BigQuery: integer range partitioning support for BQ tables

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #8645:
URL: https://github.com/apache/airflow/issues/8645#issuecomment-621836578


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org