You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "lopezvit (via GitHub)" <gi...@apache.org> on 2024/04/27 10:40:38 UTC
[I] Create a Cloud Storage Operator that could return a list of objects in a folder [airflow]
lopezvit opened a new issue, #39290:
URL: https://github.com/apache/airflow/issues/39290
### Description
Create a new operator inside of `airflow.providers.google.cloud.operators.gcs` that, given a pattern/preffix would return a list
of files in said folder, similar to the client method:
storage_client.get_bucket(BUCKET_NAME)
bucket.list_blobs(prefix=filename)
### Use case/motivation
_No response_
### Related issues
I have a process that runs once a day that reads some *.csv files from storage and process them.
It would be nice to have an operator that would do exactly that, without needing to create custom code for it.
When I create the custom code, probably using the storage hook, I can try to paste it here, but I don't have time to create a PR.
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Create a Cloud Storage Operator that could return a list of objects in a folder [airflow]
Posted by "lopezvit (via GitHub)" <gi...@apache.org>.
lopezvit commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2080457087
Ok, after a bit of googleing I found that it exists!!!
[GCSListObjectsOperator](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/operators/gcs/index.html#airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator)
Then, what I believe it is wrong is the documentation, because I didn't found it here:
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/gcs.html
Should I create a new issue for fixing the documentation or can it continue here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Create a Cloud Storage Operator that could return a list of objects in a folder [airflow]
Posted by "chaityacshah (via GitHub)" <gi...@apache.org>.
chaityacshah commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2141370471
I can take this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add GCSListObjectsOperator to operators listed in documentation in Google Provider for GCS [airflow]
Posted by "Srabasti (via GitHub)" <gi...@apache.org>.
Srabasti commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2372774191
Looks to me that the action item is to find the exact place in repository for this documentation link https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/gcs.html exists and just add the details from (https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/operators/gcs/index.html#airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator) and add section of GCSListObjectsOperator.
Carried out changes and submitted PR for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add GCSListObjectsOperator to operators listed in documentation in Google Provider for GCS [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2566684694
We have tag for it `howto_operator_gcs_list_bucket` in the code but docs do not reference the tag:
https://github.com/eladkal/airflow/blob/cd1e9f5a0488ec53b382f1f639d80a3eba8ae307/providers/tests/system/google/cloud/gcs/example_gcs_copy_delete.py#L82-L84
We need to add doc entry similar to:
https://github.com/apache/airflow/blob/3256e432d718fdd121c8fcf5687eec972e0c3f6c/docs/apache-airflow-providers-google/operators/cloud/gcs.rst#L39-L53
By the way the same case goes for `GCSDeleteObjectsOperator`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add GCSListObjectsOperator to operators listed in documentation in Google Provider for GCS [airflow]
Posted by "adan-shahid (via GitHub)" <gi...@apache.org>.
adan-shahid commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2663089627
@potiuk @eladkal Hi, If nobody is working on this issue, Kindly assign it to me. I am interested in working on this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Create a Cloud Storage Operator that could return a list of objects in a folder [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2081158158
You can just edit this Issue to request that this Operator be added to the list of operators in the docs, I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add GCSListObjectsOperator to operators listed in documentation in Google Provider for GCS [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #39290: Add GCSListObjectsOperator to operators listed in documentation in Google Provider for GCS
URL: https://github.com/apache/airflow/issues/39290
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org