[GitHub] [libcloud] Germandrummer92 commented on a diff in pull request #1847: optimize read_in_chunks

[GitBox <gi...@apache.org>]

Germandrummer92 commented on code in PR #1847:
URL: https://github.com/apache/libcloud/pull/1847#discussion_r1072754536


##########
libcloud/utils/files.py:
##########
@@ -75,7 +75,13 @@ def read_in_chunks(iterator, chunk_size=None, fill_size=False, yield_empty=False
             return
 
         if fill_size:
-            if empty or len(data) >= chunk_size:
+            chunk_start = 0

Review Comment:
   not sure if it makes it better just as an idea, but to encapsulate the comment also you could move it into a method now?
   
   ```python
   def _optimized_chunked_generator(data: bytes, chunk_size: int) -> Generator[bytes, None, bytes]:
          # We want to emit chunk_size large chunks, but chunk_size can be larger or smaller than the chunks returned
          # by get_data. We need to yield in a loop to avoid large amounts of data piling up.
          # The loop also avoids copying all data #chunks amount of times by keeping the original data as is.
          chunk_start = 0
          while chunk_start + chunk_size < len(data):
                   yield data[chunk_start : chunk_start + chunk_size]
                   chunk_start += chunk_size
                  data = data[chunk_start:]
   
        return data
   
   ...
   
   data = yield from _optimized_chunk_generator(data=data, chunk_size=chunk_size)
   ```
   
   might make this method itself a tiny bit more readable?
             



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org