You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "A-Costa (via GitHub)" <gi...@apache.org> on 2023/09/06 14:27:05 UTC
[GitHub] [airflow] A-Costa opened a new pull request, #34137: Add use_glob to GCSObjectExistenceSensor
A-Costa opened a new pull request, #34137:
URL: https://github.com/apache/airflow/pull/34137
Initial draft implementing #32650
Add the boolean parameter `use_glob` to `GCSObjectExistenceSensor`.
When set to `True` the sensor will use the `hook.list` method leveraging the newly introduced `match_glob` parameter to check if files are present, making it possible to define a`GCSObjectExistenceSensor` that checks if files that satisfy the given glob are present in a certain bucket.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1830979851
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] A-Costa commented on pull request #34137: Add use_glob to GCSObjectExistenceSensor
Posted by "A-Costa (via GitHub)" <gi...@apache.org>.
A-Costa commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1709608581
Sure, that was just an initial draft. I will try to push a complete version of this PR by the end of the day
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1859265547
Is there a minimum version of gcloud-aio needed for that ? if so we should set >= MIN
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1833433722
(plus you have still WIP in name - usualy that is an indication that this is Work In Progress - I usually just skip over such PRs even if they are rebased by author, because they are well, Work in Progress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] A-Costa commented on a diff in pull request #34137: Add use_glob to GCSObjectExistenceSensor
Posted by "A-Costa (via GitHub)" <gi...@apache.org>.
A-Costa commented on code in PR #34137:
URL: https://github.com/apache/airflow/pull/34137#discussion_r1318321804
##########
airflow/providers/google/cloud/triggers/gcs.py:
##########
@@ -98,9 +102,14 @@ async def _object_exists(self, hook: GCSAsyncHook, bucket_name: str, object_name
async with ClientSession() as s:
client = await hook.get_storage_client(s)
bucket = client.get_bucket(bucket_name)
- object_response = await bucket.blob_exists(blob_name=object_name)
- if object_response:
- return "success"
+ if self.use_glob:
+ object_response = await bucket.list_blobs(match_glob=object_name)
Review Comment:
this depends on https://github.com/talkiq/gcloud-aio/pull/634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] A-Costa commented on pull request #34137: WIP Add use_glob to GCSObjectExistenceSensor
Posted by "A-Costa (via GitHub)" <gi...@apache.org>.
A-Costa commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1718018377
https://github.com/talkiq/gcloud-aio/pull/634 was merged today so this can be fixed and merged.
Apart from the missing tests, what else needs to be fixed? @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "A-Costa (via GitHub)" <gi...@apache.org>.
A-Costa commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1859290045
Yes, the `match_glob` parameter is first exposed in the `9.0.0` release:
https://github.com/talkiq/gcloud-aio/releases/tag/storage-9.0.0
I'll add the minimum version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #34137: WIP Add use_glob to GCSObjectExistenceSensor
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1713629229
Needs fixes though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1833431231
Generallly - making PR green is a prerequisite for anyone to take further look - if tests are failing and you were told to fix them and nothing happened, well, then the ball is on your side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "A-Costa (via GitHub)" <gi...@apache.org>.
A-Costa commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1833532496
> [talkiq/gcloud-aio#634](https://github.com/talkiq/gcloud-aio/pull/634) was merged today so this can be fixed and merged.
>
> Apart from the missing tests, what else needs to be fixed? @potiuk
Here i asked if something else needed fix
> > Apart from the missing tests, what else needs to be fixed?
>
> I think that is all but we will take closer look in 2nd review once everything is ready You'll need also to bump `gcloud-aio` version in `provider.yaml` so it will include the fix in the upstream library
Here a 2nd review was mentioned that never arrived.
> Needs fixes though.
this was your only message so far, not very helpful to a new contributor.
Will work on the things you pointed out now though...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1762776491
> Apart from the missing tests, what else needs to be fixed?
I think that is all but we will take closer look in 2nd review once everything is ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1833566575
> Here i asked if something else needed fix
Well. The CI tells you what needs to be fixed. I think you should take a look at the contributor's guide and understand that fixing and rebasing CI is what is the "first pass". You have one PR to submit. We regularly review 100s of PR a day. And CI and tests are the way to automate thing that must be fixed before we do next pass of reviews. So my comment - while it had been expressed with brevity - reflect the state -of matter. you need to fix tests before we move any furtther.
In case it is not clear - it's all described in our contributor's guide:
* How to communicate (where it is written that it's author's responsiblity to mae sure their PR gets reviewed - including periodoc rebasing and followin up on it) https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#how-to-communicate
* How to prepare PR: https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#step-4-prepare-pr
* How to iterate on PR: https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#step-5-pass-pr-review
I understand it's a lot of information to digest, so I perfectly understand your lack of understanding how our PR process looks like and what is expected from author, but now that you got it explained in a little more than "tests need fixing" I hope you will continue contributing and follow those guidelines.
I think also what helps is understand that it's you who want to contribute something to the project, so it's a good idea to pay attention to your own PR and if in doubt consult the documentation.
Looking forward to your future contributions. No hard feelings from my side, I hope with those explanations you will understand why the ball was on your side all the time, you just did not follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "A-Costa (via GitHub)" <gi...@apache.org>.
A-Costa commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1833368893
any update on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1833428875
> any update on this?
Well. The PR is waiting for your actions:
> Needs fixes though
> I think that is all but we will take closer look in 2nd review once everything is ready
> You'll need also to bump gcloud-aio version in provider.yaml so it will include the fix in the upstream library
Tests are failed and @eladkal asked you to update libraries.
Why do you think you need any update before following these comments ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "A-Costa (via GitHub)" <gi...@apache.org>.
A-Costa commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1859217555
I've added the missing test for GCSBlobTrigger and removed WIP from the title, as this is ready to be reviewed.
> You'll need also to bump `gcloud-aio` version in `provider.yaml` so it will include the fix in the upstream library
@eladkal regarding this, it seems to me that the version is not pinned in the yaml, so it should be fine already: https://github.com/apache/airflow/blob/f7f71836175b81484fe6afb147a58e1ca6d00f4d/airflow/providers/google/provider.yaml#L90
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #34137:
URL: https://github.com/apache/airflow/pull/34137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #34137: Add use_glob to GCSObjectExistenceSensor
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1708480712
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #34137: Add use_glob to GCSObjectExistenceSensor
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1709404067
Can you please add unit tests for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1833609121
Apologies if MY comments were also seen as condescent (which I realise they might) - but this is simply coming from the sheer volume of things we have to deal with - it's very difficult to balance brevity (not helpful) with long messages (TLDR) and the time it needs to put, so we rely a lot on the docs we put together on contributing, but - yeah - it's difficult to follow, I understand that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "A-Costa (via GitHub)" <gi...@apache.org>.
A-Costa commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1833595976
@potiuk of course no hard feelings on my side either! I appreciate the exchange and I'm sorry if I came up as rude, was never the intention!
Now that is more clear what is expected, and that action is required on my side, I'll be working on the things that are missing! Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1833566157
Happy to review when you mark the PR as ready :)
Sometimes we do multiple runs of reviews that is very OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add use_glob to GCSObjectExistenceSensor [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #34137:
URL: https://github.com/apache/airflow/pull/34137#issuecomment-1859306225
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org