You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/30 19:45:03 UTC
[GitHub] [airflow] exploy opened a new pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables
exploy opened a new pull request #8653:
URL: https://github.com/apache/airflow/pull/8653
Support for integer range partitioning for BigQuery tables
There is already support for time partitioning while creating BQ tables and Google offers more options than only time partitioning.
We found integer range partitioning working best for our case so we customized few classes to gain this functionality.
https://github.com/apache/airflow/issues/8416
---
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Target Github ISSUE in description if exists
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
mik-laj closed pull request #8653:
URL: https://github.com/apache/airflow/pull/8653
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #8653:
URL: https://github.com/apache/airflow/pull/8653#issuecomment-638749967
We have a different approach to this operator now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #8653:
URL: https://github.com/apache/airflow/pull/8653#issuecomment-623308491
Hi @exploy , you can use `table_resource` argument to pass any table definition you want, no need to specify every single parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #8653:
URL: https://github.com/apache/airflow/pull/8653#issuecomment-623208822
Hi. Integration with BigQuery is undergoing extensive refactor by @turbaszek . Can you wait until we finish this work? It is possible that then this change will not be necessary because these operators will accept all parameters available through the API.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] exploy commented on pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
exploy commented on pull request #8653:
URL: https://github.com/apache/airflow/pull/8653#issuecomment-629149784
> Hi @exploy , you can use `table_resource` argument to pass any table definition you want, no need to specify every single parameter.
Does this mean that time partitioning args will be removed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BhuviTheDataGuy commented on pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
BhuviTheDataGuy commented on pull request #8653:
URL: https://github.com/apache/airflow/pull/8653#issuecomment-717322389
In the BQ operator which parameter I can use to mention the partition?
`api_resource_configs ` or `query_params`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BhuviTheDataGuy commented on pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
BhuviTheDataGuy commented on pull request #8653:
URL: https://github.com/apache/airflow/pull/8653#issuecomment-717688893
Never mind, I figured it out. It is `query_params`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
turbaszek commented on pull request #8653:
URL: https://github.com/apache/airflow/pull/8653#issuecomment-630640207
> > Hi @exploy , you can use `table_resource` argument to pass any table definition you want, no need to specify every single parameter.
>
> Does this mean that time partitioning args will be removed?
Yes, our intention is to minimize the operator interface. In this way there will be no problem with missing fields and the code will be easier to maintain
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BhuviTheDataGuy removed a comment on pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables
Posted by GitBox <gi...@apache.org>.
BhuviTheDataGuy removed a comment on pull request #8653:
URL: https://github.com/apache/airflow/pull/8653#issuecomment-717688893
Never mind, I figured it out. It is `query_params`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org