[jira] [Commented] (IMPALA-12358) Skip scheduling runtime filter that unlikely to meet RUNTIME_FILTER_WAIT_TIME_MS constraint

["David Rorke (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867447#comment-17867447 ] 

David Rorke commented on IMPALA-12358:
--------------------------------------

We should be able to develop a general costing model for the filters that takes into account various factors including:
 * The expected selectivity of the filter (becomes trickier for cascading filters)
 * Expected reduction of scanning time based on the selectivity (some of the recent changes for compute processing cost, e.g. IMPALA-12657 already attempt to do scan cost predictions by combining cardinality estimates with baseline benchmark results)

We could use this "expected filter benefit" both to determine the set of filters to generate at planning time and also the amount of time we should wait for an individual filter (we might still schedule a filter that's only expected to reduce scan time by 1 second but we definitely wouldn't wait 10 seconds for it).

> Skip scheduling runtime filter that unlikely to meet RUNTIME_FILTER_WAIT_TIME_MS constraint
> -------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-12358
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12358
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Frontend
>            Reporter: Riza Suminto
>            Priority: Major
>              Labels: runtime-filters
>
> A scan node will wait for a runtime filter arrival for [RUNTIME_FILTER_WAIT_TIME_MS|https://impala.apache.org/docs/build/html/topics/impala_runtime_filter_wait_time_ms.html]. If the runtime filter is not arrived within this period, the scan node will stop waiting and start scanning. Late runtime filter can still be applied if the scanning has not finished yet upon filter arrival. However, planner should predict to some degree if a certain runtime filter is likely to be late. If it does, then skip scheduling such a filter.
> The prediction can be based on either of:
>  * total volume of all scans from the build side of join.
>  * the distance between the join node and the furthest fragment in the build side direction.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org