[I] `NestedLoopsJoin` memory tracking insufficient [arrow-datafusion]

["alamb (via GitHub)" <gi...@apache.org>]

alamb opened a new issue, #8952:
URL: https://github.com/apache/arrow-datafusion/issues/8952

   ### Is your feature request related to a problem or challenge?
   
   Similarly to https://github.com/apache/arrow-datafusion/issues/7848, @metesynnada noted  https://github.com/apache/arrow-datafusion/pull/8020#issuecomment-1903359773 that it is possible for `NestedLoopsJoin` to generate a single (very) large `RecordBatch`. For certain pathalogical queries this may lead to DataFusion far exceeding its memory limits and erroring out
   
   ### Describe the solution you'd like
   
   Implement / adapt the same approach as @korowa  did in https://github.com/apache/arrow-datafusion/pull/8020 (❤️ ) to incrementally create join output for joins that match many keys rather than doing it all at once. 
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] `NestedLoopsJoin` memory tracking may be insufficient [arrow-datafusion]

["yyy1000 (via GitHub)" <gi...@apache.org>]

yyy1000 commented on issue #8952:
URL: https://github.com/apache/arrow-datafusion/issues/8952#issuecomment-1915867842

   I'd like a try to help it. :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org