[I] [Bug] Benchmark sink configuration loading leaves directory streams open [paimon]

["taoran92 (via GitHub)" <gi...@apache.org>]

taoran92 opened a new issue, #10137:
URL: https://github.com/apache/paimon/issues/10137

   ### Search before asking
   
   - [x] I searched in the issues and found nothing similar.
   
   ### Paimon version
   
   master at `ad1e6c187817f23c2e20c15f6c59d5923fc5b6b0` (2.2-SNAPSHOT).
   
   ### Compute Engine
   
   Standalone Java benchmark configuration loading in `paimon-cluster-benchmark`; no running Flink job is required. The underlying directory-stream behavior was reproduced on macOS with JDK 8.
   
   ### Minimal reproduce step
   
   1. Create a temporary directory containing a `sinks` subdirectory. An empty `sinks` directory is sufficient to exercise the directory enumeration.
   2. Call `org.apache.paimon.benchmark.Sink.load(location)` repeatedly in the same JVM.
   3. Observe that the directory stream opened on each invocation is never explicitly closed.
   
   The relevant code is:
   
   ```java
   Iterator<Path> it = Files.list(sinkLocation).iterator();
   while (it.hasNext()) {
       // Read and parse sink configuration files.
   }
   ```
   
   A separate JDK 8 probe of this exact API usage (`Files.list(...).iterator()` followed by complete iteration) showed 20 additional open file descriptors after 20 iterations. Explicitly closing the retained streams returned the descriptor count to its baseline.
   
   ### What doesn't meet your expectations?
   
   `Files.list` returns a resource-owning stream. Exhausting its iterator does not close the underlying directory stream, and `Sink.load` discards the stream reference without closing it. The directory resource is therefore left open after normal completion, as well as when configuration-file reading or YAML parsing throws an exception.
   
   The stream should be closed on both normal and exceptional exits. Repeated calls can accumulate directory resources. The current benchmark startup normally loads sink configurations once, so this is a small resource-management defect rather than evidence of widespread benchmark failures.
   
   ### Anything else?
   
   Use try-with-resources around the `Stream<Path>` returned by `Files.list`, keeping the entire existing iterator loop inside the block. This preserves configuration-loading behavior and requires no new dependencies.
   
   Affected file: `paimon-benchmark/paimon-cluster-benchmark/src/main/java/org/apache/paimon/benchmark/Sink.java`.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Benchmark sink configuration loading leaves directory streams open [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi closed issue #10137: [Bug] Benchmark sink configuration loading leaves directory streams open
URL: https://github.com/apache/paimon/issues/10137


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org