[PR] [python] Fix RecursionError when reading a shard or slice of a large file [paimon]

["jackylee-ch (via GitHub)" <gi...@apache.org>]

jackylee-ch opened a new pull request, #10113:
URL: https://github.com/apache/paimon/pull/10113

   ### Purpose
   
   `ShardBatchReader.read_arrow_batch` skips each out-of-range batch by recursively calling itself, so recursion depth grows one per skipped batch. A `with_shard`/`with_slice` read whose range sits deep in a data file then overflows the stack with `RecursionError`: the pyarrow reader yields 1024-row batches by default, so a slice starting ~1M rows in skips >1000 batches (Python's recursion limit). A slice covering only the head of a large file hits it too — the batches after `end_pos` are drained the same recursive way.
   
   The fix iterates over skipped batches with a `while` loop, matching `ConcatBatchReader` and `ApplyDeletionVectorReader`. Behavior is otherwise unchanged.
   
   ### Tests
   
   `shard_batch_reader_test`: 2000 single-row batches, slice `(1999, 2000)` — fails on master with `RecursionError`, passes here. Two more cases cover range filtering (`[2,5)`) and the straddle branches (`[2,9)` over 4-row batches).
   
   Written with Claude Code; verification is mine.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [python] Fix RecursionError when reading a shard or slice of a large file [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #10113:
URL: https://github.com/apache/paimon/pull/10113


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [python] Fix RecursionError when reading a shard or slice of a large file [paimon]

["Akash3121 (via GitHub)" <gi...@apache.org>]

Akash3121 commented on code in PR #10113:
URL: https://github.com/apache/paimon/pull/10113#discussion_r4089745739


##########
paimon-python/pypaimon/read/reader/shard_batch_reader.py:
##########
@@ -38,8 +38,14 @@ def read_arrow_batch(self) -> Optional[RecordBatch]:
         if isinstance(self.reader.format_reader, FormatBlobReader):
             # For blob reader, pass begin_idx and end_idx parameters
             return self.reader.read_arrow_batch(start_idx=self.start_pos, end_idx=self.end_pos)
-        else:
-            # For non-blob reader (DataFileBatchReader), use standard read_arrow_batch
+
+        # For non-blob reader (DataFileBatchReader), use standard read_arrow_batch.
+        # Loop rather than recurse over skipped batches: a slice/shard whose range
+        # sits deep in a file (default parquet batch_size is 1024 rows) skips one
+        # batch per step, so recursing here overflows the stack (RecursionError)
+        # once the skipped count exceeds the interpreter limit. Mirrors the
+        # while-loop skip pattern in ConcatBatchReader / ApplyDeletionVectorReader.
+        while True:

Review Comment:
   The loop removes the recursion failure, but it still drains every batch after  `end_pos` . Callers such as  `TableRead.to_arrow()`  read until this method returns  `None` , so for a  `[0, 1)`  slice of a 2,000-batch file, the first call returns row 0 and the second call reads the remaining 1,999 batches plus EOF. This means a head slice still scans the entire file, and an error in data outside the selected range is surfaced even though that data should never be read. Please return  `None`  before calling the underlying reader once  `current_pos >= end_pos`  (or track an exhausted flag). Add a counting/raising reader test for a large  `[0, 1)`  slice and assert that the second call returns  `None`  without another underlying read.
   
   Reproduction: a counting reader with 2,000 one-row batches required 1 call to return  `[0]` , then reached 2,001 total calls before the patched reader returned  `None` . A reader configured to throw on its second batch returned the selected first row and then incorrectly propagated  `OSError: tail should not be read`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org