[jira] [Created] (ARROW-15519) [C++] Investigate potential performance improvments for the filter node

["Weston Pace (Jira)" <ji...@apache.org>]

Weston Pace created ARROW-15519:
-----------------------------------

             Summary: [C++] Investigate potential performance improvments for the filter node
                 Key: ARROW-15519
                 URL: https://issues.apache.org/jira/browse/ARROW-15519
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++
            Reporter: Weston Pace


Right now some early runs with Arrowbench and the OT PR (https://github.com/apache/arrow/pull/12100) shows that we spend a fair amount of time in TPC-H queries on filter nodes.  There are a few improvements we know could be made to our filtering approach at the moment.  I'm creating this parent issue to help categorize and track those:

 * We can use a selection vector in our filters to reduce the amount of materialization needed.  While long term we may want to support a selection vector throughout the exec plan a good start would be to use it when we encounter a chain of filters to avoid excess materialization (e.g. x < 10 && x > 5 && y < 20)
 * If a filter if very selective then we may end up outputting a lot of very small batches.  We could probably hold onto the data at the filter node until we've accumulated enough rows for a decent sized batch.
 * The filter node is currently creating new thread tasks instead of appending its work onto an existing thread task.
 * If we have a chain of filters we could potentially use runtime selectivity statistics / estimates to reorder our filters so that the most selective filters are evaluated first.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)