[I] Core: Avoid fixed polling delays in ParallelIterable for fast manifest reads [iceberg]

["fanruan-tuco (via GitHub)" <gi...@apache.org>]

fanruan-tuco opened a new issue, #18521:
URL: https://github.com/apache/iceberg/issues/18521

   ### Apache Iceberg version
   
   main (development).
   
   ### Query engine
   
   Other — Java API reproduction.
   
   ### Please describe the bug 🐞
   
   Scan planning is slow with large numbers of files, even with fast manifest reads:
   
   - `HadoopCatalog` + `HadoopFileIO` on local disk.
   - S3 + a caching `FileIO`, with manifest cache hits.
   
   When the result queue is empty and tasks are pending, `ParallelIterable.hasNext()` sleeps for 10 ms. Results arriving or tasks completing during that sleep do not wake the consumer. Repeated polling delays can dominate fast reads.
   
   In a synthetic test with 1,000 one-element tasks, 10 threads, 20 task slots, and a 1 ms delayed start per task, the original took about 505 ms (50 sleeps) versus 54 ms with notification-based waiting, approximately 9.4× faster (medians of 10 runs after 2 warm-ups; not an end-to-end storage benchmark).
   
   Could we wake the consumer on data arrival, completion, failure, or closure instead of polling?
   
   <details>
   <summary>Minimal reproduction against the original implementation (imports omitted)</summary>
   
   ```java
   public class ParallelIterablePollingRepro {
     public static void main(String[] args) throws Exception {
       System.setProperty("iceberg.worker.num-threads", "10");
       ScheduledThreadPoolExecutor workers = new ScheduledThreadPoolExecutor(10) {
         @Override
         public void execute(Runnable command) {
           schedule(command, 1, TimeUnit.MILLISECONDS);
         }
       };
       workers.prestartAllCoreThreads();
   
       try {
         List<Iterable<Integer>> inputs = Collections.nCopies(1000, List.of(1));
         long start = System.nanoTime();
         int count = 0;
         try (ParallelIterable<Integer> results = new ParallelIterable<>(inputs, workers)) {
           for (Integer ignored : results) {
             count++;
           }
         }
         System.out.printf("items=%d elapsed=%.3f ms%n", count,
             (System.nanoTime() - start) / 1_000_000.0);
       } finally {
         workers.shutdownNow();
         workers.awaitTermination(5, TimeUnit.SECONDS);
       }
     }
   }
   ```
   
   </details>
   
   Related: [#10691 discussion](https://github.com/apache/iceberg/pull/10691#discussion_r1685429472), [#4596 discussion](https://github.com/apache/iceberg/pull/4596#discussion_r882301640).
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] Core: Avoid fixed polling delays in ParallelIterable for fast manifest reads [iceberg]

["fanruan-tuco (via GitHub)" <gi...@apache.org>]

fanruan-tuco commented on issue #18521:
URL: https://github.com/apache/iceberg/issues/18521#issuecomment-6207806296

   I would like to fix it, PR: https://github.com/apache/iceberg/pull/18522


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org