You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "lopezvit (via GitHub)" <gi...@apache.org> on 2024/04/27 10:40:38 UTC

[I] Create a Cloud Storage Operator that could return a list of objects in a folder [airflow]

lopezvit opened a new issue, #39290:
URL: https://github.com/apache/airflow/issues/39290

   ### Description
   
   Create a new operator inside of `airflow.providers.google.cloud.operators.gcs` that, given a pattern/preffix would return a list 
   of files in said folder, similar to the client method:
   
       storage_client.get_bucket(BUCKET_NAME)
       bucket.list_blobs(prefix=filename)
   
   ### Use case/motivation
   
   _No response_
   
   ### Related issues
   
   I have a process that runs once a day that reads some *.csv files from storage and process them.
   It would be nice to have an operator that would do exactly that, without needing to create custom code for it.
   When I create the custom code, probably using the storage hook, I can try to paste it here, but I don't have time to create a PR.
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Create a Cloud Storage Operator that could return a list of objects in a folder [airflow]

Posted by "lopezvit (via GitHub)" <gi...@apache.org>.
lopezvit commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2080457087

   Ok, after a bit of googleing I found that it exists!!!
   [GCSListObjectsOperator](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/operators/gcs/index.html#airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator)
   Then, what I believe it is wrong is the documentation, because I didn't found it here:
   https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/gcs.html
   Should I create a new issue for fixing the documentation or can it continue here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Create a Cloud Storage Operator that could return a list of objects in a folder [airflow]

Posted by "chaityacshah (via GitHub)" <gi...@apache.org>.
chaityacshah commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2141370471

   I can take this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add GCSListObjectsOperator to operators listed in documentation in Google Provider for GCS [airflow]

Posted by "Srabasti (via GitHub)" <gi...@apache.org>.
Srabasti commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2372774191

   Looks to me that the action item is to find the exact place in repository for this documentation link https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/gcs.html exists and just add the details from (https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/operators/gcs/index.html#airflow.providers.google.cloud.operators.gcs.GCSListObjectsOperator) and add section of GCSListObjectsOperator. 
   Carried out changes and submitted PR for review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add GCSListObjectsOperator to operators listed in documentation in Google Provider for GCS [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2566684694

   We have tag for it `howto_operator_gcs_list_bucket`  in the code but docs do not reference the tag:
   
   https://github.com/eladkal/airflow/blob/cd1e9f5a0488ec53b382f1f639d80a3eba8ae307/providers/tests/system/google/cloud/gcs/example_gcs_copy_delete.py#L82-L84
   
   We need to add doc entry similar to:
   
   https://github.com/apache/airflow/blob/3256e432d718fdd121c8fcf5687eec972e0c3f6c/docs/apache-airflow-providers-google/operators/cloud/gcs.rst#L39-L53
   
   By the way the same case goes for `GCSDeleteObjectsOperator`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add GCSListObjectsOperator to operators listed in documentation in Google Provider for GCS [airflow]

Posted by "adan-shahid (via GitHub)" <gi...@apache.org>.
adan-shahid commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2663089627

   @potiuk  @eladkal   Hi, If nobody is working on this issue, Kindly assign it to me. I am interested in working on this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Create a Cloud Storage Operator that could return a list of objects in a folder [airflow]

Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #39290:
URL: https://github.com/apache/airflow/issues/39290#issuecomment-2081158158

   You can just edit this Issue to request that this Operator be added to the list of operators in the docs, I think.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add GCSListObjectsOperator to operators listed in documentation in Google Provider for GCS [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #39290: Add GCSListObjectsOperator to operators listed in documentation in Google Provider for GCS 
URL: https://github.com/apache/airflow/issues/39290


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org