You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/07 13:06:44 UTC
[GitHub] [airflow] thejens opened a new pull request #10218: Implement Google BigQuery Table Partition Sensor
thejens opened a new pull request #10218:
URL: https://github.com/apache/airflow/pull/10218
With this change we implement a new sensor that checks the existence of a partition in a partitioned BigQuery table.
Currently you can check whether a table exists with the BigQueryTableExistenceSensor, however a common use case is for tasks to append partitions to a table in daily batch executions, or to stream data into hourly/date/int-range partitions.
A partition is a common artifact from jobs and systems, and it is a common use case to want to ensure that a partition exists before reading data from it downstream, and with this change you can, using BigQueryTablePartitionExistenceSensor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10218: Implement Google BigQuery Table Partition Sensor
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10218:
URL: https://github.com/apache/airflow/pull/10218#issuecomment-673475886
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #10218: Implement Google BigQuery Table Partition Sensor
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #10218:
URL: https://github.com/apache/airflow/pull/10218#issuecomment-670506760
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thejens edited a comment on pull request #10218: Implement Google BigQuery Table Partition Sensor
Posted by GitBox <gi...@apache.org>.
thejens edited a comment on pull request #10218:
URL: https://github.com/apache/airflow/pull/10218#issuecomment-671329600
@potiuk Thanks for the feedback, I added the pre-commit hook, worked out well.
I think I got everything right now, had to rebase after some faulty merge
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10218: Implement Google BigQuery Table Partition Sensor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10218:
URL: https://github.com/apache/airflow/pull/10218#issuecomment-673477460
What's next :) ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10218: Implement Google BigQuery Table Partition Sensor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10218:
URL: https://github.com/apache/airflow/pull/10218#issuecomment-673476029
Thanks @thejens . This is fantastic contribution!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thejens commented on pull request #10218: Implement Google BigQuery Table Partition Sensor
Posted by GitBox <gi...@apache.org>.
thejens commented on pull request #10218:
URL: https://github.com/apache/airflow/pull/10218#issuecomment-671259789
Hi, I've added a new example dag file, though I'd be open to add to the already rather convoluted example_bigquery_queries , should I also add it to the integration test_bigquery_system.py file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thejens edited a comment on pull request #10218: Implement Google BigQuery Table Partition Sensor
Posted by GitBox <gi...@apache.org>.
thejens edited a comment on pull request #10218:
URL: https://github.com/apache/airflow/pull/10218#issuecomment-671329600
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10218: Implement Google BigQuery Table Partition Sensor
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10218:
URL: https://github.com/apache/airflow/pull/10218#issuecomment-671280073
Looks good. I think it's ok to keep separate example dag. One thing do - it would be great to extract parts of it to the HowTo Guide (there are those comments that are used to include some parts of the example: https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/example_dags/example_dataproc.py#L45
Plus some static checks are failing :(
Heartily recommend pre-commit installation https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#pre-commit-hooks - it saves a TON of time for back/forth with CI :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #10218: Implement Google BigQuery Table Partition Sensor
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #10218:
URL: https://github.com/apache/airflow/pull/10218
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] thejens commented on pull request #10218: Implement Google BigQuery Table Partition Sensor
Posted by GitBox <gi...@apache.org>.
thejens commented on pull request #10218:
URL: https://github.com/apache/airflow/pull/10218#issuecomment-671329600
@potiuk Thanks for the feedback, I added the pre-commit hook, worked out well.
I think I got everything right now, but might need some help deciphering test errors if this current build doesn't work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org