[GitHub] [druid] suneet-s commented on a change in pull request #10312: Optimize large InDimFilters

[GitBox <gi...@apache.org>]

suneet-s commented on a change in pull request #10312:
URL: https://github.com/apache/druid/pull/10312#discussion_r475354734



##########
File path: processing/src/main/java/org/apache/druid/query/filter/InDimFilter.java
##########
@@ -143,10 +142,11 @@ private InDimFilter(
 
     // The values set can be huge. Try to avoid copying the set if possible.
     // Note that we may still need to copy values to a list for caching. See getCacheKey().
-    if ((NullHandling.sqlCompatible() || values.stream().noneMatch(NullHandling::needsEmptyToNull))) {
+    if (NullHandling.sqlCompatible() || !values.remove("")) {
       this.values = values;
     } else {
-      this.values = values.stream().map(NullHandling::emptyToNullIfNeeded).collect(Collectors.toSet());
+      values.add(null);

Review comment:
       I think the if condition on line 145 works such that it only enters this else block if there was an empty string already in the set. I should add unit tests here, because I had to think a lot about how the ordering of the if statements affected the code flow 😅 and how nulls and empty strings work in the different modes...
   
   What I want to happen here is:
   * If it's sqlCompatible mode, just use the values as is
   * If it's default mode (ie not sql compatible mode) attempt to remove empty string
   * If an empty string was removed add null
   * If no empty  string was removed, use values as is




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org