You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hussein-awala (via GitHub)" <gi...@apache.org> on 2023/12/09 15:23:07 UTC
Re: [PR] GCSHook only perform list if delimiter specified [airflow]
hussein-awala commented on code in PR #36130:
URL: https://github.com/apache/airflow/pull/36130#discussion_r1421434993
##########
airflow/providers/google/cloud/hooks/gcs.py:
##########
@@ -821,7 +821,8 @@ def _list(
delimiter=delimiter,
versions=versions,
)
- list(blobs)
+ if delimiter:
+ list(blobs)
Review Comment:
IMHO, there is a problem with the original PR (#34919) that added this statement, where we create the generators to avoid loading all the data in the memory, but by applying `list` on the blobs generator, it looks like you are forcing the generator to loop over all the pages and load them in memory.
Instead, the solution should be based on a loop on the generator elements similar to what we do here:
https://github.com/apache/airflow/blob/5d74ffb32095d534866f029d085198bc783d82c2/airflow/providers/google/cloud/hooks/gcs.py#L941-L945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org