Re: [PR] IN LIST: add branchless filter for small primitive lists [datafusion]

["kosiew (via GitHub)" <gi...@apache.org>]

kosiew commented on code in PR #23014:
URL: https://github.com/apache/datafusion/pull/23014#discussion_r3549435092


##########
datafusion/physical-expr/src/expressions/in_list/strategy.rs:
##########
@@ -54,8 +124,60 @@ pub(super) fn instantiate_static_filter(
         DataType::Float32 => Ok(Arc::new(Float32StaticFilter::try_new(&in_array)?)),
         DataType::Float64 => Ok(Arc::new(Float64StaticFilter::try_new(&in_array)?)),
         _ => {
-            /* fall through to generic implementation for unsupported types (Struct, etc.) */
+            // Fall through to generic implementation for unsupported types
+            // (Struct, etc.).
             Ok(Arc::new(ArrayStaticFilter::try_new(in_array)?))
         }
     }
 }
+
+fn bitmap_filter<T>(in_array: &ArrayRef) -> Result<StaticFilterRef>
+where
+    T: BitmapFilterType,
+{
+    Ok(Arc::new(BitmapFilter::<T>::try_new(in_array)?))
+}
+
+fn branchless_filter<T>(
+    in_array: &ArrayRef,
+    non_null_count: usize,
+) -> Result<Option<StaticFilterRef>>
+where
+    T: BranchlessFilterType,
+    BranchlessNative<T>: Copy + PartialEq + Send + Sync,
+{
+    if non_null_count > T::MAX_LIST_LEN {

Review Comment:
   Nice improvement here. The new branchless dispatch looks well covered through the direct `BranchlessFilter` tests.
   
   As a small follow-up, it might be worth adding one strategy-level test around the cutoff behavior, for example `MAX_LIST_LEN` vs `MAX_LIST_LEN + 1`, and possibly an all-null or zero-non-null list. That would help keep the `instantiate_static_filter` routing covered if these thresholds or fallbacks change later. Non-blocking.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org