[I] NestedLoopJoin memory-limited fallback re-executes the build side, losing rows for non-replayable inputs [datafusion]

["ranflarion (via GitHub)" <gi...@apache.org>]

ranflarion opened a new issue, #24676:
URL: https://github.com/apache/datafusion/issues/24676

   ### Describe the bug
   
   `NestedLoopJoinExec`'s memory-limited fallback re-executes the build-side child. When the in-memory load fails with `ResourcesExhausted`, `initiate_fallback` calls `self.left.execute(0, ctx)` a second time and spills that stream to disk — but the first execution already consumed part (or all) of the input. For an input that cannot be executed twice, the batches the first pass consumed are gone:
   
   - If the second execution yields nothing, the fallback fails with `Internal("Left side produced no data to spill")`.
   - If the second execution resumes a shared stream (e.g. an `ExecutionPlan` wrapping an external one-shot reader — FFI sources, engines embedding DataFusion that feed operators from a network or JVM iterator), the join silently returns only the rows the first pass had not yet consumed. We observed this embedding DataFusion 54.1.0: a 200k-row build side against an 8MB pool returned 134,464 rows with no error.
   
   Even for replayable inputs the build side is fully executed twice, paying the child's cost again just to write it to disk.
   
   ### To Reproduce
   
   Unit-shape repro against `main`, in `nested_loop_join.rs`'s test module: wrap the existing `build_left_table()` in an `ExecutionPlan` whose `execute` returns the real stream on the first call and an empty stream afterwards, then run any join through `join_collect` with a 50-byte memory limit (as `test_nlj_memory_limited_inner_join` does). The fallback engages, re-executes the child, gets the empty stream, and fails:
   
   ```
   Internal("Left side produced no data to spill")
   ```
   
   With a stream that resumes instead of restarting, the same shape loses the consumed prefix silently.
   
   ### Expected behavior
   
   The fallback should spill from the pass that already consumed the stream: when a reservation fails, write the batches buffered so far plus the remainder of the same stream to the spill file, instead of executing the child again. This executes the build side exactly once whether or not it fits, works for non-replayable inputs, and removes the double execution for replayable ones. `SpillState::Pending` then no longer needs to carry the child plan.
   
   We run this design in production against DataFusion 54.1.0 and can offer it as a PR.
   
   ### Additional context
   
   Related: #15760 (memory-limited NLJ), #22038 (cross-partition left bitmap), #23076 (concat memory amplification in the same load path).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] NestedLoopJoin memory-limited fallback re-executes the build side, losing rows for non-replayable inputs [datafusion]

["jayzhan211 (via GitHub)" <gi...@apache.org>]

jayzhan211 closed issue #24676: NestedLoopJoin memory-limited fallback re-executes the build side, losing rows for non-replayable inputs
URL: https://github.com/apache/datafusion/issues/24676


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org