[GitHub] [libcloud] RunOrVeith commented on a diff in pull request #1847: optimize read_in_chunks

[GitBox <gi...@apache.org>]

RunOrVeith commented on code in PR #1847:
URL: https://github.com/apache/libcloud/pull/1847#discussion_r1072344773


##########
libcloud/utils/files.py:
##########
@@ -75,7 +75,13 @@ def read_in_chunks(iterator, chunk_size=None, fill_size=False, yield_empty=False
             return
 
         if fill_size:
-            if empty or len(data) >= chunk_size:
+            chunk_start = 0

Review Comment:
   maybe add a quick explanation why it is smarter to do it this way than the old way



##########
libcloud/test/test_utils.py:
##########
@@ -202,15 +202,42 @@ def iterator():
             for x in range(0, 1000):
                 yield "aa"
 
+        chunk_count = 0
         for result in libcloud.utils.files.read_in_chunks(
             iterator(), chunk_size=10, fill_size=False
         ):
+            chunk_count += 1
             self.assertEqual(result, b("aa"))
+        self.assertEqual(chunk_count, 1000)
 
+        chunk_count = 0
         for result in libcloud.utils.files.read_in_chunks(
             iterator(), chunk_size=10, fill_size=True
         ):
+            chunk_count += 1
             self.assertEqual(result, b("aaaaaaaaaa"))
+        self.assertEqual(chunk_count, 200)
+
+    def test_read_in_chunks_large_iterator_batches(self):
+        def iterator():
+            for x in range(0, 10):
+                yield "a" * 10_000
+
+        chunk_count = 0
+        for result in libcloud.utils.files.read_in_chunks(
+            iterator(), chunk_size=10, fill_size=False
+        ):
+            chunk_count += 1
+            self.assertEqual(result, b("a") * 10_000)
+        self.assertEqual(chunk_count, 10)
+
+        chunk_count = 0
+        for result in libcloud.utils.files.read_in_chunks(
+            iterator(), chunk_size=10, fill_size=True
+        ):
+            chunk_count += 1
+            self.assertEqual(result, b("aaaaaaaaaa"))

Review Comment:
   If you make this b("a") * 10 it's clearer to see what is going on



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org