[GitHub] [iceberg] lirui-apache commented on a diff in pull request #4596: Use bounded queue to avoid consuming too much memory

[GitBox <gi...@apache.org>]

lirui-apache commented on code in PR #4596:
URL: https://github.com/apache/iceberg/pull/4596#discussion_r882301640


##########
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##########
@@ -65,10 +67,10 @@ private ParallelIterator(Iterable<? extends Iterable<T>> iterables,
             try (Closeable ignored = (iterable instanceof Closeable) ?
                 (Closeable) iterable : () -> { }) {
               for (T item : iterable) {
-                queue.add(item);
+                queue.put(item);
               }
-            } catch (IOException e) {
-              throw new RuntimeIOException(e, "Failed to close iterable");
+            } catch (IOException | InterruptedException e) {
+              throw new RuntimeException("Failed to close iterable", e);

Review Comment:
   BTW, I think we shouldn't call `Thread.sleep` in `hasNext` if we use blocking queue. In our test it's bad for performance when the queue size is small.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org