You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Weston Pace (Jira)" <ji...@apache.org> on 2022/02/02 00:33:00 UTC

[jira] [Created] (ARROW-15521) [C++] Investigate batching filter node output

Weston Pace created ARROW-15521:
-----------------------------------

             Summary: [C++] Investigate batching filter node output
                 Key: ARROW-15521
                 URL: https://issues.apache.org/jira/browse/ARROW-15521
             Project: Apache Arrow
          Issue Type: Sub-task
          Components: C++
            Reporter: Weston Pace


A highly selective filter could emit tiny batches.  Running with tiny batches we have a lot of overhead.

On the other hand, merging batches together requires allocation and copies so it comes with overhead of its own.

When things are more stable we should investigate whether there is some threshold where we merge batches together after a filter.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)