(spark) branch branch-4.1 updated: [SPARK-57958][SQL][TEST] Close the ORC Reader in OrcSourceSuite to fix leaked file streams

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.1 by this push:
     new d0fe0b3507ce [SPARK-57958][SQL][TEST] Close the ORC Reader in OrcSourceSuite to fix leaked file streams
d0fe0b3507ce is described below

commit d0fe0b3507cebcfad4d44d9db05814fbee4c87fd
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Mon Jul 6 18:53:18 2026 +0900

    [SPARK-57958][SQL][TEST] Close the ORC Reader in OrcSourceSuite to fix leaked file streams
    
    ### What changes were proposed in this pull request?
    In `OrcSuite.testBloomFilterCreation` and `testSelectiveDictionaryEncoding`, also close the ORC `Reader` (returned by `OrcFile.createReader`) in the `finally` block — previously only the derived `RecordReader` was closed.
    
    ### Why are the changes needed?
    Both helpers do:
    ```scala
    val reader = OrcFile.createReader(orcFilePath, readerOptions)
    var recordReader: RecordReaderImpl = null
    try {
      recordReader = reader.rows.asInstanceOf[RecordReaderImpl]
      ...
    } finally {
      if (recordReader != null) recordReader.close()   // Reader itself is never closed
    }
    ```
    The `Reader` holds the file stream opened in `org.apache.orc.impl.ReaderImpl.extractFileTail`, so it leaks. `SharedSparkSessionBase.afterEach` asserts no open streams (`DebugFilesystem.assertNoOpenStreams`) and aborts the whole suite when the leaked stream is not closed within its wait window:
    
    ```
    org.apache.spark.sql.execution.datasources.orc.OrcSourceV1Suite *** ABORTED ***
      ... There are 1 possibly leaked file streams. (SharedSparkSession.scala:197)
    ```
    The leaked-connection stack (logged by `DebugFilesystem`) points at `org.apache.orc.impl.ReaderImpl.extractFileTail` / `OrcFile.createReader`. This flakes `OrcSourceV1Suite`/`OrcSourceV2Suite` (observed repeatedly on `build_java21` for branch-4.0). Closing the `Reader` fixes the leak.
    
    ### Does this PR introduce _any_ user-facing change?
    No, test-only.
    
    ### How was this patch tested?
    Ran `OrcSourceV1Suite` and `OrcSourceV2Suite` on GitHub Actions.
    
    - **Before (failing on `apache/spark` `build_java21` / branch-4.0):** https://github.com/apache/spark/actions/runs/28766468775/job/85293800991 — `OrcSourceV1Suite *** ABORTED ***`, `There are 1 possibly leaked file streams. (SharedSparkSession.scala:197)`.
    - **After (passing with this change):** https://github.com/HyukjinKwon/spark-agent6/actions/runs/28781272553/job/85337180343 — `OrcSourceV1Suite` + `OrcSourceV2Suite`: `Tests: succeeded 72, failed 0`, `All tests passed.` (no leaked-stream abort).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Yes.
    
    This pull request and its description were written by Isaac.
    
    Closes #57036 from HyukjinKwon/ci-fix/agent5-orc-reader-leak.
    
    Authored-by: Hyukjin Kwon <gu...@apache.org>
    Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
    (cherry picked from commit 9633f932aec024e067238b8df9ef16491bd4df4d)
    Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
---
 .../org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
index 040999476ece..0b039e2fdc78 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala
@@ -121,6 +121,7 @@ abstract class OrcSuite
           if (recordReader != null) {
             recordReader.close()
           }
+          reader.close()
         }
       }
     }
@@ -191,6 +192,7 @@ abstract class OrcSuite
           if (recordReader != null) {
             recordReader.close()
           }
+          reader.close()
         }
       }
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org