You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "clintropolis (via GitHub)" <gi...@apache.org> on 2024/03/05 05:34:45 UTC

[PR] add new typed in filter (druid)

clintropolis opened a new pull request, #16039:
URL: https://github.com/apache/druid/pull/16039

   ### Description
   _real description TBD_
   
   changes:
   * adds TypedInFilter which preserves matching sets in the native match value type
   * SQL planner uses new TypedInFilter when druid.generic.useDefaultValueForNull=false (the default)
   
   #### Release note
   _TBD_
   
   <hr>
   
   This PR has:
   
   - [ ] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534873891


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {

Review Comment:
   reworked this to only compute if sortedValues is not lazy



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534864533


##########
processing/src/main/java/org/apache/druid/segment/index/semantic/ValueSetIndexes.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.index.semantic;
+
+import com.google.common.base.Supplier;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.collections.bitmap.BitmapFactory;
+import org.apache.druid.collections.bitmap.ImmutableBitmap;
+import org.apache.druid.segment.column.TypeSignature;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.SimpleImmutableBitmapDelegatingIterableIndex;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+
+public interface ValueSetIndexes
+{
+  double SORTED_MERGE_RATIO_THRESHOLD = 0.12D;
+  int SIZE_WORTH_CHECKING_MIN = 8;
+
+  /**
+   * Get the wrapped {@link ImmutableBitmap} corresponding to the specified set of values (if they are contained in the
+   * underlying column). The set must be sorted using the comparator of the supplied matchValueType.
+   *
+   * @param sortedValues   values to match, sorted in matchValueType order
+   * @param matchValueType type of the value to match, used to assist conversion from the match value type to the column
+   *                       value type
+   * @return               {@link ImmutableBitmap} corresponding to the rows which match the values, or null if an index
+   *                       connot be computed for the supplied value type
+   */
+  @Nullable
+  BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType);
+
+  static <T> BitmapColumnIndex getIndexFromSortedIteratorSortedMerged(

Review Comment:
   renamed stuff and added javadocs to explain the relation between these three functions and when to use them



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1530892864


##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/ArrayOverlapOperatorConversion.java:
##########
@@ -127,7 +130,7 @@ public DimFilter toDruidFilter(
         // to create an empty array with no argument, we just return null.
         return null;
       } else if (arrayElements.length == 1) {
-        if (plannerContext.isUseBoundsAndSelectors()) {
+        if (plannerContext.isUseBoundsAndSelectors() || (!simpleExtractionExpr.isDirectColumnAccess() && virtualColumnRegistry == null)) {

Review Comment:
   when is `virtualColumnRegistry` null?
   
   & if it can be null, then I think CodeQL makes a good point: could it be null in the `else` branch as well, where there is no `null` guard?



##########
processing/src/main/java/org/apache/druid/segment/index/semantic/ValueSetIndexes.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.index.semantic;
+
+import com.google.common.base.Supplier;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.collections.bitmap.BitmapFactory;
+import org.apache.druid.collections.bitmap.ImmutableBitmap;
+import org.apache.druid.segment.column.TypeSignature;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.SimpleImmutableBitmapDelegatingIterableIndex;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+
+public interface ValueSetIndexes
+{
+  double SORTED_MERGE_RATIO_THRESHOLD = 0.12D;
+  int SIZE_WORTH_CHECKING_MIN = 8;
+
+  /**
+   * Get the wrapped {@link ImmutableBitmap} corresponding to the specified set of values (if they are contained in the
+   * underlying column). The set must be sorted using the comparator of the supplied matchValueType.
+   *
+   * @param sortedValues   values to match, sorted in matchValueType order
+   * @param matchValueType type of the value to match, used to assist conversion from the match value type to the column
+   *                       value type
+   * @return               {@link ImmutableBitmap} corresponding to the rows which match the values, or null if an index
+   *                       connot be computed for the supplied value type
+   */
+  @Nullable
+  BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType);
+
+  static <T> BitmapColumnIndex getIndexFromSortedIteratorSortedMerged(

Review Comment:
   Javadoc for this and `getIndexFromSortedIterator`? For some reason, the names don't make it obvious to me what they do. (One is regular and one is `SortedMerged`?)
   
   Reading the code, it looks like this one does a linear scan through the dictionary, and the other one does a series of binary searches. Maybe call them `getIndexFromSortedIteratorWithScan` and `getIndexFromSortedIteratorWithBinarySearch`.
   
   Oh, now I see why it's called `SortedMerged`. The zipping is kind of like the merge step of a merge-sort. Still, I feel the "scan" and "binary search" names would make more sense. And Javadoc would definitely help.



##########
processing/src/main/java/org/apache/druid/segment/index/semantic/ValueSetIndexes.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.index.semantic;
+
+import com.google.common.base.Supplier;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.collections.bitmap.BitmapFactory;
+import org.apache.druid.collections.bitmap.ImmutableBitmap;
+import org.apache.druid.segment.column.TypeSignature;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.SimpleImmutableBitmapDelegatingIterableIndex;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+
+public interface ValueSetIndexes
+{
+  double SORTED_MERGE_RATIO_THRESHOLD = 0.12D;
+  int SIZE_WORTH_CHECKING_MIN = 8;
+
+  /**
+   * Get the wrapped {@link ImmutableBitmap} corresponding to the specified set of values (if they are contained in the
+   * underlying column). The set must be sorted using the comparator of the supplied matchValueType.
+   *
+   * @param sortedValues   values to match, sorted in matchValueType order
+   * @param matchValueType type of the value to match, used to assist conversion from the match value type to the column
+   *                       value type
+   * @return               {@link ImmutableBitmap} corresponding to the rows which match the values, or null if an index
+   *                       connot be computed for the supplied value type
+   */
+  @Nullable
+  BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType);
+
+  static <T> BitmapColumnIndex getIndexFromSortedIteratorSortedMerged(
+      BitmapFactory bitmapFactory,
+      Comparator<T> comparator,
+      Iterable<T> values,
+      Indexed<T> dictionary,
+      Indexed<ImmutableBitmap> bitmaps,
+      Supplier<ImmutableBitmap> unknownsBitmap
+  )
+  {
+    return new BaseValueSetIndexesFromIterable(bitmapFactory, bitmaps, unknownsBitmap)
+    {
+      @Override
+      public Iterable<ImmutableBitmap> getBitmapIterable()
+      {
+        return () -> new Iterator<ImmutableBitmap>()
+        {
+          final PeekingIterator<T> valuesIterator = Iterators.peekingIterator(values.iterator());
+          final PeekingIterator<T> dictionaryIterator = Iterators.peekingIterator(dictionary.iterator());
+          int next = -1;
+          int idx = 0;
+
+          @Override
+          public boolean hasNext()
+          {
+            if (next < 0) {
+              findNext();
+            }
+            return next >= 0;
+          }
+
+          @Override
+          public ImmutableBitmap next()
+          {
+            if (next < 0) {
+              findNext();
+              if (next < 0) {
+                throw new NoSuchElementException();
+              }
+            }
+            final int swap = next;
+            next = -1;
+            return getBitmap(swap);
+          }
+
+          private void findNext()
+          {
+            while (next < 0 && valuesIterator.hasNext() && dictionaryIterator.hasNext()) {
+              final T nextValue = valuesIterator.peek();
+              final T nextDictionaryKey = dictionaryIterator.peek();
+              final int comparison = comparator.compare(nextValue, nextDictionaryKey);
+              if (comparison == 0) {
+                next = idx;
+                valuesIterator.next();
+                break;
+              } else if (comparison < 0) {
+                valuesIterator.next();
+              } else {
+                dictionaryIterator.next();
+                idx++;
+              }
+            }
+          }
+        };
+      }
+    };
+  }
+
+  static <T> BitmapColumnIndex getIndexFromSortedIterator(
+      BitmapFactory bitmapFactory,
+      Iterable<T> values,
+      Indexed<T> dictionary,
+      Indexed<ImmutableBitmap> bitmaps,
+      Supplier<ImmutableBitmap> getUnknownsIndex
+  )
+  {
+    return new BaseValueSetIndexesFromIterable(bitmapFactory, bitmaps, getUnknownsIndex)
+    {
+      @Override
+      public Iterable<ImmutableBitmap> getBitmapIterable()
+      {
+        return () -> new Iterator<ImmutableBitmap>()
+        {
+          final int dictionarySize = dictionary.size();
+          final Iterator<T> iterator = values.iterator();
+          int next = -1;
+
+          @Override
+          public boolean hasNext()
+          {
+            if (next < 0) {
+              findNext();
+            }
+            return next >= 0;
+          }
+
+          @Override
+          public ImmutableBitmap next()
+          {
+            if (next < 0) {
+              findNext();
+              if (next < 0) {
+                throw new NoSuchElementException();
+              }
+            }
+            final int swap = next;
+            next = -1;
+            return getBitmap(swap);
+          }
+
+          private void findNext()
+          {
+            while (next < 0 && iterator.hasNext()) {
+              T nextValue = iterator.next();
+              next = dictionary.indexOf(nextValue);
+
+              if (next == -dictionarySize - 1) {
+                // nextValue is past the end of the dictionary so we can break early
+                // Note: we can rely on indexOf returning (-(insertion point) - 1), because of the earlier check
+                // for Indexed.isSorted(), which guarantees this behavior
+                break;
+              }
+            }
+          }
+        };
+      }
+    };
+  }
+
+  static <T> BitmapColumnIndex getIndexFromIterator(

Review Comment:
   Javadoc would be useful for this one too.



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());
+        for (Object s : lazyMatchValues.get()) {
+          matchValueBytes.add(StringUtils.toUtf8ByteBuffer(Evals.asString(s)));
+        }
+        return matchValueBytes;
+      });
+    } else {
+      this.lazyMatchValueBytes = null;
+    }
+
+    this.predicateFactorySupplier = Suppliers.memoize(
+        () -> new PredicateFactory(lazyMatchValues.get(), matchValueType)
+    );
+    this.cacheKeySupplier = Suppliers.memoize(this::computeCacheKey);
+  }
+
+  @JsonProperty
+  public String getColumn()
+  {
+    return column;
+  }
+
+  @JsonProperty
+  public List<?> getSortedValues()
+  {
+    return lazyMatchValues.get();
+  }
+
+  @JsonProperty
+  public ColumnType getMatchValueType()
+  {
+    return matchValueType;
+  }
+
+  @Nullable
+  @JsonInclude(JsonInclude.Include.NON_NULL)
+  @JsonProperty
+  public FilterTuning getFilterTuning()
+  {
+    return filterTuning;
+  }
+
+  @Override
+  public byte[] getCacheKey()
+  {
+    return cacheKeySupplier.get();
+  }
+
+  @Override
+  public DimFilter optimize(final boolean mayIncludeUnknown)
+  {
+    final List<?> matchValues = lazyMatchValues.get();
+    if (matchValues.isEmpty()) {
+      return FalseDimFilter.instance();
+    } else if (matchValues.size() == 1) {
+      if (matchValues.get(0) == null) {
+        return NullFilter.forColumn(column);
+      }
+      return new EqualityFilter(
+          column,
+          matchValueType,
+          matchValues.iterator().next(),
+          filterTuning
+      );
+    }
+    return this;
+  }
+
+  @Override
+  public Filter toFilter()
+  {
+    return this;
+  }
+
+  @Nullable
+  @Override
+  public RangeSet<String> getDimensionRangeSet(String dimension)
+  {
+    if (!Objects.equals(getColumn(), dimension)) {
+      return null;
+    }
+    RangeSet<String> retSet = TreeRangeSet.create();
+    for (Object value : lazyMatchValues.get()) {
+      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));
+      if (valueEquivalent == null) {
+        // Case when SQL compatible null handling is enabled
+        // Range.singleton(null) is invalid, so use the fact that
+        // only null values are less than empty string.
+        retSet.add(Range.lessThan(""));
+      } else {
+        retSet.add(Range.singleton(valueEquivalent));
+      }
+    }
+    return retSet;
+  }
+
+  @Override
+  public Set<String> getRequiredColumns()
+  {
+    return ImmutableSet.of(column);
+  }
+
+  @Override
+  @Nullable
+  public BitmapColumnIndex getBitmapColumnIndex(ColumnIndexSelector selector)
+  {
+    if (!Filters.checkFilterTuningUseIndex(column, selector, filterTuning)) {
+      return null;
+    }
+    final ColumnIndexSupplier indexSupplier = selector.getIndexSupplier(column);
+
+    if (indexSupplier == null) {
+      // column doesn't exist, match against null
+      DruidPredicateMatch match = predicateFactorySupplier.get().makeStringPredicate().apply(null);
+      return Filters.makeMissingColumnNullIndex(match, selector);
+    }
+
+    if (lazyMatchValueBytes != null) {
+      final Utf8ValueSetIndexes utf8ValueSetIndexes = indexSupplier.as(Utf8ValueSetIndexes.class);
+      if (utf8ValueSetIndexes != null) {
+        return utf8ValueSetIndexes.forSortedValuesUtf8(lazyMatchValueBytes.get());
+      }
+    }
+
+    final ValueSetIndexes valueSetIndexes = indexSupplier.as(ValueSetIndexes.class);
+    if (valueSetIndexes != null) {
+      return valueSetIndexes.forSortedValues(lazyMatchValues.get(), matchValueType);
+    }
+
+    return Filters.makePredicateIndex(
+        column,
+        selector,
+        predicateFactorySupplier.get()
+    );
+  }
+
+  @Override
+  public ValueMatcher makeMatcher(ColumnSelectorFactory factory)
+  {
+    return Filters.makeValueMatcher(factory, column, predicateFactorySupplier.get());
+  }
+
+  @Override
+  public VectorValueMatcher makeVectorMatcher(final VectorColumnSelectorFactory factory)
+  {
+    return ColumnProcessors.makeVectorProcessor(
+        column,
+        VectorValueMatcherColumnProcessorFactory.instance(),
+        factory
+    ).makeMatcher(predicateFactorySupplier.get());
+  }
+
+  @Override
+  public boolean canVectorizeMatcher(ColumnInspector inspector)
+  {
+    return true;
+  }
+
+  @Override
+  public boolean supportsRequiredColumnRewrite()
+  {
+    return true;
+  }
+
+  @Override
+  public Filter rewriteRequiredColumns(Map<String, String> columnRewrites)
+  {
+    String rewriteDimensionTo = columnRewrites.get(column);
+    if (rewriteDimensionTo == null) {
+      throw new IAE("Received a non-applicable rewrite: %s, filter's dimension: %s", columnRewrites, column);
+    }
+
+    if (rewriteDimensionTo.equals(column)) {
+      return this;
+    } else {
+      return new TypedInFilter(
+          rewriteDimensionTo,
+          matchValueType,
+          null,
+          lazyMatchValues.get(),
+          filterTuning
+      );
+    }
+  }
+
+  @Override
+  public String toString()
+  {
+    final DimFilter.DimFilterToStringBuilder builder = new DimFilter.DimFilterToStringBuilder();
+    return builder.appendDimension(column, null)
+                  .append(" IN (")
+                  .append(Joiner.on(", ").join(Iterables.transform(lazyMatchValues.get(), String::valueOf)))
+                  .append(")")
+                  .append(" (" + matchValueType + ")")
+                  .appendFilterTuning(filterTuning)
+                  .build();
+  }
+
+  @Override
+  public boolean equals(Object o)
+  {
+    if (this == o) {
+      return true;
+    }
+    if (o == null || getClass() != o.getClass()) {
+      return false;
+    }
+    TypedInFilter that = (TypedInFilter) o;
+    return column.equals(that.column) &&
+           Objects.equals(matchValueType, that.matchValueType) &&
+           compareValues(lazyMatchValues.get(), that.lazyMatchValues.get(), matchValueType) &&
+           Objects.equals(filterTuning, that.filterTuning);
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Objects.hash(lazyMatchValues.get(), column, matchValueType, filterTuning);
+  }
+
+  private byte[] computeCacheKey()
+  {
+    // Hash all values, in sorted order, as their length followed by their content.
+    final Hasher hasher = Hashing.sha256().newHasher();
+    for (Object v : lazyMatchValues.get()) {
+      if (v == null) {
+        // Encode null as length -1, no content.
+        hasher.putInt(-1);
+      } else {
+        final String s = Evals.asString(v);
+        hasher.putInt(s.length());
+        hasher.putString(s, StandardCharsets.UTF_8);
+      }
+    }
+
+    return new CacheKeyBuilder(DimFilterUtils.NEW_IN_CACHE_ID)
+        .appendString(column)
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendString(matchValueType.asTypeString())
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendByteArray(hasher.hash().asBytes())
+        .build();
+  }
+
+  private static boolean checkSorted(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Comparator<Object> comparator = matchValueType.getNullableStrategy();
+    Object prev = null;
+    boolean needsCoerceCheck = true;
+    for (Object o : unsortedValues) {
+      if (needsCoerceCheck && o != null) {
+        Object coerced = coerceValue(o, matchValueType);
+        //noinspection ObjectEquality
+        if (coerced != o) {
+          return false;
+        }
+        needsCoerceCheck = false;
+      }
+      if (prev != null && comparator.compare(prev, o) >= 0) {
+        return false;
+      }
+      prev = o;
+    }
+    return true;
+  }
+
+  @Nullable
+  private static Object coerceValue(@Nullable Object o, ColumnType matchValueType)
+  {
+    if (o == null) {
+      return o;
+    }
+    switch (matchValueType.getType()) {
+      case STRING:
+        return DimensionHandlerUtils.convertObjectToString(o);
+      case LONG:
+        return DimensionHandlerUtils.convertObjectToLong(o);
+      case FLOAT:
+        return DimensionHandlerUtils.convertObjectToFloat(o);
+      case DOUBLE:
+        return DimensionHandlerUtils.convertObjectToDouble(o);
+      default:
+        throw InvalidInput.exception("Unsupported matchValueType[%s]", matchValueType);
+    }
+  }
+
+  private static List<?> sortValues(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Object[] array = unsortedValues.toArray(new Object[0]);
+    // check if values need coerced
+    for (int i = 0; i < array.length; i++) {
+      Object coerced = coerceValue(array[i], matchValueType);
+      //noinspection ObjectEquality
+      if (coerced != null && array[i] == coerced) {
+        // assume list is all same type objects...

Review Comment:
   Is this going to cause issues when reading JSON that has `values` set rather than `sortedValues`? (Where they might not all be the same type)
   
   Also, do we care? I suppose if we aren't documenting this, then there's no way we'd get JSON with `values` in it. The serializer always writes `sortedValues`. You know, though, if this is the thinking, then we should really also split the constructor into two: a `@JsonCreator` that takes `sortedValues` only (not `values`) and a non-Jackson-enabled constructor that accepts possibly-unsorted values. No sense in supporting `values` in the JSON if we aren't going to use it.



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());

Review Comment:
   Couldn't this be a `List` rather than a tree set? The values are already in the right order thanks to `lazyMatchValues.get()`.



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {

Review Comment:
   Do we need to check for duplicates too, or are duplicates ok in `lazyMatchValues`? Similar question for `sortValues`: does it need to dedupe or is just sorting ok?



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;

Review Comment:
   Some Javadoc about this and the others (`lazyMatchValues`, `lazyMatchValueBytes`) would be helpful, explaining the invariants in play. Reading the rest of the code, I think it's something like: 
   
   - `lazyMatchValues` is always set and always returns a set sorted by `matchValueType`
   - `unsortedValues` is set if we know that the values backing `lazyMatchValues` haven't been sorted yet
   - regardless of the above, when we serialize via JSON, the values are always sorted and always set under `sortedValues`. this is then trusted at deserialization time.
   
   What about duplicates, though? Might `unsortedValues` or `lazyMatchValues` contain duplicates?



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {

Review Comment:
   If `lazyMatchValues` came in through `sortedValues` (indicating we're running on a data server or running with known-sorted data) then I think this should be eager too.
   
   Reason for that is if it's _not_ eager, probably it will get computed in some processing thread while processing some segment, and then all other processing threads for the same query will block on it. (They'll likely all be trying to compute this at the same time.)
   
   Probably still want it to be lazy if `checkSorted` returns `true` but `sortedValues` was not set. That's likely to happen during SQL planning, and there's no need to do utf8 encoding at that time.



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter

Review Comment:
   Some Javadoc for this and `InDimFilter` that point to each other would be good. People new to this area of the code base will need to be informed that both exist.



##########
processing/src/main/java/org/apache/druid/segment/nested/ScalarDoubleColumnAndIndexSupplier.java:
##########
@@ -258,6 +265,70 @@ public <T> T computeBitmapResult(BitmapResultFactory<T> bitmapResultFactory, boo
     }
   }
 
+  private final class DoubleValueSetIndexes implements ValueSetIndexes
+  {
+    final FixedIndexed<Double> dictionary = doubleDictionarySupplier.get();
+
+    @Nullable
+    @Override
+    public BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType)
+    {
+      final boolean matchNull = sortedValues.get(0) == null;
+      final Supplier<ImmutableBitmap> unknownsIndex = () -> {
+        if (!matchNull && dictionary.get(0) == null) {
+          return valueIndexes.get(0);
+        }
+        return null;
+      };
+
+      // values are doubles and ordered in double order
+      if (matchValueType.is(ValueType.DOUBLE)) {

Review Comment:
   Is it possible to minimize the copied code between here and the long version, such as by using a shared helper somewhere?



##########
processing/src/test/java/org/apache/druid/segment/filter/TypedInFilterTests.java:
##########
@@ -0,0 +1,509 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.filter;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Function;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.RangeSet;
+import nl.jqno.equalsverifier.EqualsVerifier;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.query.filter.DimFilter;
+import org.apache.druid.query.filter.Filter;
+import org.apache.druid.query.filter.NotDimFilter;
+import org.apache.druid.query.filter.TypedInFilter;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.StorageAdapter;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.Closeable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+@RunWith(Enclosed.class)
+public class TypedInFilterTests

Review Comment:
   Is there a way to write a test class that subclasses `InFilterTest` so it automatically gets any new tests we add there? Maybe make `inFilter` a `protected abstract` method rather than a `static`. The copied tests here make it tough to properly maintain the test suite.



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(

Review Comment:
   Better to do this check during `toFilter()` or one of the runtime methods instead of at dimFilter construction time, since doing it at the `@JsonCreator` path means that this filter can't be deserialized if the flag changes value. That might cause issues since filters end up in task specs (`transformSpec`) etc.



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());
+        for (Object s : lazyMatchValues.get()) {
+          matchValueBytes.add(StringUtils.toUtf8ByteBuffer(Evals.asString(s)));
+        }
+        return matchValueBytes;
+      });
+    } else {
+      this.lazyMatchValueBytes = null;
+    }
+
+    this.predicateFactorySupplier = Suppliers.memoize(
+        () -> new PredicateFactory(lazyMatchValues.get(), matchValueType)
+    );
+    this.cacheKeySupplier = Suppliers.memoize(this::computeCacheKey);
+  }
+
+  @JsonProperty
+  public String getColumn()
+  {
+    return column;
+  }
+
+  @JsonProperty
+  public List<?> getSortedValues()
+  {
+    return lazyMatchValues.get();
+  }
+
+  @JsonProperty
+  public ColumnType getMatchValueType()
+  {
+    return matchValueType;
+  }
+
+  @Nullable
+  @JsonInclude(JsonInclude.Include.NON_NULL)
+  @JsonProperty
+  public FilterTuning getFilterTuning()
+  {
+    return filterTuning;
+  }
+
+  @Override
+  public byte[] getCacheKey()
+  {
+    return cacheKeySupplier.get();
+  }
+
+  @Override
+  public DimFilter optimize(final boolean mayIncludeUnknown)
+  {
+    final List<?> matchValues = lazyMatchValues.get();
+    if (matchValues.isEmpty()) {
+      return FalseDimFilter.instance();
+    } else if (matchValues.size() == 1) {
+      if (matchValues.get(0) == null) {
+        return NullFilter.forColumn(column);
+      }
+      return new EqualityFilter(
+          column,
+          matchValueType,
+          matchValues.iterator().next(),
+          filterTuning
+      );
+    }
+    return this;
+  }
+
+  @Override
+  public Filter toFilter()
+  {
+    return this;
+  }
+
+  @Nullable
+  @Override
+  public RangeSet<String> getDimensionRangeSet(String dimension)
+  {
+    if (!Objects.equals(getColumn(), dimension)) {
+      return null;
+    }
+    RangeSet<String> retSet = TreeRangeSet.create();
+    for (Object value : lazyMatchValues.get()) {
+      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));
+      if (valueEquivalent == null) {
+        // Case when SQL compatible null handling is enabled
+        // Range.singleton(null) is invalid, so use the fact that
+        // only null values are less than empty string.
+        retSet.add(Range.lessThan(""));
+      } else {
+        retSet.add(Range.singleton(valueEquivalent));
+      }
+    }
+    return retSet;
+  }
+
+  @Override
+  public Set<String> getRequiredColumns()
+  {
+    return ImmutableSet.of(column);
+  }
+
+  @Override
+  @Nullable
+  public BitmapColumnIndex getBitmapColumnIndex(ColumnIndexSelector selector)
+  {
+    if (!Filters.checkFilterTuningUseIndex(column, selector, filterTuning)) {
+      return null;
+    }
+    final ColumnIndexSupplier indexSupplier = selector.getIndexSupplier(column);
+
+    if (indexSupplier == null) {
+      // column doesn't exist, match against null
+      DruidPredicateMatch match = predicateFactorySupplier.get().makeStringPredicate().apply(null);
+      return Filters.makeMissingColumnNullIndex(match, selector);
+    }
+
+    if (lazyMatchValueBytes != null) {
+      final Utf8ValueSetIndexes utf8ValueSetIndexes = indexSupplier.as(Utf8ValueSetIndexes.class);
+      if (utf8ValueSetIndexes != null) {
+        return utf8ValueSetIndexes.forSortedValuesUtf8(lazyMatchValueBytes.get());
+      }
+    }
+
+    final ValueSetIndexes valueSetIndexes = indexSupplier.as(ValueSetIndexes.class);
+    if (valueSetIndexes != null) {
+      return valueSetIndexes.forSortedValues(lazyMatchValues.get(), matchValueType);
+    }
+
+    return Filters.makePredicateIndex(
+        column,
+        selector,
+        predicateFactorySupplier.get()
+    );
+  }
+
+  @Override
+  public ValueMatcher makeMatcher(ColumnSelectorFactory factory)
+  {
+    return Filters.makeValueMatcher(factory, column, predicateFactorySupplier.get());
+  }
+
+  @Override
+  public VectorValueMatcher makeVectorMatcher(final VectorColumnSelectorFactory factory)
+  {
+    return ColumnProcessors.makeVectorProcessor(
+        column,
+        VectorValueMatcherColumnProcessorFactory.instance(),
+        factory
+    ).makeMatcher(predicateFactorySupplier.get());
+  }
+
+  @Override
+  public boolean canVectorizeMatcher(ColumnInspector inspector)
+  {
+    return true;
+  }
+
+  @Override
+  public boolean supportsRequiredColumnRewrite()
+  {
+    return true;
+  }
+
+  @Override
+  public Filter rewriteRequiredColumns(Map<String, String> columnRewrites)
+  {
+    String rewriteDimensionTo = columnRewrites.get(column);
+    if (rewriteDimensionTo == null) {
+      throw new IAE("Received a non-applicable rewrite: %s, filter's dimension: %s", columnRewrites, column);
+    }
+
+    if (rewriteDimensionTo.equals(column)) {
+      return this;
+    } else {
+      return new TypedInFilter(
+          rewriteDimensionTo,
+          matchValueType,
+          null,
+          lazyMatchValues.get(),
+          filterTuning
+      );
+    }
+  }
+
+  @Override
+  public String toString()
+  {
+    final DimFilter.DimFilterToStringBuilder builder = new DimFilter.DimFilterToStringBuilder();
+    return builder.appendDimension(column, null)
+                  .append(" IN (")
+                  .append(Joiner.on(", ").join(Iterables.transform(lazyMatchValues.get(), String::valueOf)))
+                  .append(")")
+                  .append(" (" + matchValueType + ")")
+                  .appendFilterTuning(filterTuning)
+                  .build();
+  }
+
+  @Override
+  public boolean equals(Object o)
+  {
+    if (this == o) {
+      return true;
+    }
+    if (o == null || getClass() != o.getClass()) {
+      return false;
+    }
+    TypedInFilter that = (TypedInFilter) o;
+    return column.equals(that.column) &&
+           Objects.equals(matchValueType, that.matchValueType) &&
+           compareValues(lazyMatchValues.get(), that.lazyMatchValues.get(), matchValueType) &&
+           Objects.equals(filterTuning, that.filterTuning);
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Objects.hash(lazyMatchValues.get(), column, matchValueType, filterTuning);
+  }
+
+  private byte[] computeCacheKey()
+  {
+    // Hash all values, in sorted order, as their length followed by their content.
+    final Hasher hasher = Hashing.sha256().newHasher();
+    for (Object v : lazyMatchValues.get()) {
+      if (v == null) {
+        // Encode null as length -1, no content.
+        hasher.putInt(-1);
+      } else {
+        final String s = Evals.asString(v);
+        hasher.putInt(s.length());
+        hasher.putString(s, StandardCharsets.UTF_8);
+      }
+    }
+
+    return new CacheKeyBuilder(DimFilterUtils.NEW_IN_CACHE_ID)
+        .appendString(column)
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendString(matchValueType.asTypeString())
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendByteArray(hasher.hash().asBytes())
+        .build();
+  }
+
+  private static boolean checkSorted(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Comparator<Object> comparator = matchValueType.getNullableStrategy();
+    Object prev = null;
+    boolean needsCoerceCheck = true;
+    for (Object o : unsortedValues) {
+      if (needsCoerceCheck && o != null) {
+        Object coerced = coerceValue(o, matchValueType);
+        //noinspection ObjectEquality
+        if (coerced != o) {
+          return false;
+        }
+        needsCoerceCheck = false;
+      }
+      if (prev != null && comparator.compare(prev, o) >= 0) {
+        return false;
+      }
+      prev = o;
+    }
+    return true;
+  }
+
+  @Nullable
+  private static Object coerceValue(@Nullable Object o, ColumnType matchValueType)
+  {
+    if (o == null) {
+      return o;
+    }
+    switch (matchValueType.getType()) {
+      case STRING:
+        return DimensionHandlerUtils.convertObjectToString(o);
+      case LONG:
+        return DimensionHandlerUtils.convertObjectToLong(o);
+      case FLOAT:
+        return DimensionHandlerUtils.convertObjectToFloat(o);
+      case DOUBLE:
+        return DimensionHandlerUtils.convertObjectToDouble(o);
+      default:
+        throw InvalidInput.exception("Unsupported matchValueType[%s]", matchValueType);
+    }
+  }
+
+  private static List<?> sortValues(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Object[] array = unsortedValues.toArray(new Object[0]);
+    // check if values need coerced
+    for (int i = 0; i < array.length; i++) {
+      Object coerced = coerceValue(array[i], matchValueType);
+      //noinspection ObjectEquality
+      if (coerced != null && array[i] == coerced) {
+        // assume list is all same type objects...
+        break;
+      }
+      array[i] = coerced;
+    }
+    ObjectArrays.quickSort(array, matchValueType.getNullableStrategy());
+    return Arrays.asList(array);
+  }
+
+  /**
+   * Since jackson might translate longs into ints and such, we use the type comparator to check lists
+   * for {@link #equals(Object)} instead of directly using {@link Objects#equals(Object, Object)}
+   */
+  private static boolean compareValues(List<?> o1, List<?> o2, ColumnType matchValueType)
+  {
+    final NullableTypeStrategy comparator = matchValueType.getNullableStrategy();
+    //noinspection ObjectEquality
+    if (o1 == o2) {
+      return true;
+    }
+    if (o1 == null) {
+      return false;
+    }
+    if (o2 == null) {
+      return false;
+    }
+    final int iter = Math.min(o1.size(), o2.size());
+    for (int i = 0; i < iter; i++) {
+      final int cmp = comparator.compare(o1.get(i), o2.get(i));
+      if (cmp == 0) {
+        continue;
+      }
+      return false;
+    }
+    return o1.size() == o2.size();
+  }
+
+  private static DruidObjectPredicate<String> createStringPredicate(
+      final List sortedValues,

Review Comment:
   I usually prefer `List<?>` if it works ok, because it at least keeps the unknownness to the single type param rather than encouraging the dropping of all type params.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534874307


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {

Review Comment:
   added javadocs to hopefully better explain inner workings



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      this.unsortedValues = values;
+      this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());
+        for (Object s : lazyMatchValues.get()) {
+          matchValueBytes.add(StringUtils.toUtf8ByteBuffer(Evals.asString(s)));
+        }
+        return matchValueBytes;
+      });
+    } else {
+      this.lazyMatchValueBytes = null;
+    }
+
+    this.predicateFactorySupplier = Suppliers.memoize(
+        () -> new InFilterDruidPredicateFactory(lazyMatchValues.get(), matchValueType)
+    );
+    this.cacheKeySupplier = Suppliers.memoize(this::computeCacheKey);
+  }
+
+  @JsonProperty
+  public String getColumn()
+  {
+    return column;
+  }
+
+  @JsonProperty
+  public List<?> getSortedValues()
+  {
+    return lazyMatchValues.get();
+  }
+
+  @JsonProperty
+  public ColumnType getMatchValueType()
+  {
+    return matchValueType;
+  }
+
+  @Nullable
+  @JsonInclude(JsonInclude.Include.NON_NULL)
+  @JsonProperty
+  public FilterTuning getFilterTuning()
+  {
+    return filterTuning;
+  }
+
+  @Override
+  public byte[] getCacheKey()
+  {
+    return cacheKeySupplier.get();
+  }
+
+  @Override
+  public DimFilter optimize(final boolean mayIncludeUnknown)
+  {
+    final List<?> matchValues = lazyMatchValues.get();
+    if (matchValues.isEmpty()) {
+      return FalseDimFilter.instance();
+    } else if (matchValues.size() == 1) {
+      if (matchValues.get(0) == null) {
+        return NullFilter.forColumn(column);
+      }
+      return new EqualityFilter(
+          column,
+          matchValueType,
+          matchValues.iterator().next(),
+          filterTuning
+      );
+    }
+    return this;
+  }
+
+  @Override
+  public Filter toFilter()
+  {
+    return this;
+  }
+
+  @Nullable
+  @Override
+  public RangeSet<String> getDimensionRangeSet(String dimension)
+  {
+    if (!Objects.equals(getColumn(), dimension)) {
+      return null;
+    }
+    RangeSet<String> retSet = TreeRangeSet.create();
+    for (Object value : lazyMatchValues.get()) {
+      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));
+      if (valueEquivalent == null) {
+        // Case when SQL compatible null handling is enabled
+        // Range.singleton(null) is invalid, so use the fact that
+        // only null values are less than empty string.
+        retSet.add(Range.lessThan(""));
+      } else {
+        retSet.add(Range.singleton(valueEquivalent));
+      }
+    }
+    return retSet;
+  }
+
+  @Override
+  public Set<String> getRequiredColumns()
+  {
+    return ImmutableSet.of(column);
+  }
+
+  @Override
+  @Nullable
+  public BitmapColumnIndex getBitmapColumnIndex(ColumnIndexSelector selector)
+  {
+    if (!Filters.checkFilterTuningUseIndex(column, selector, filterTuning)) {
+      return null;
+    }
+    final ColumnIndexSupplier indexSupplier = selector.getIndexSupplier(column);
+
+    if (indexSupplier == null) {
+      // column doesn't exist, match against null
+      DruidPredicateMatch match = predicateFactorySupplier.get().makeStringPredicate().apply(null);
+      return Filters.makeMissingColumnNullIndex(match, selector);
+    }
+
+    if (lazyMatchValueBytes != null) {
+      final Utf8ValueSetIndexes utf8ValueSetIndexes = indexSupplier.as(Utf8ValueSetIndexes.class);
+      if (utf8ValueSetIndexes != null) {
+        return utf8ValueSetIndexes.forSortedValuesUtf8(lazyMatchValueBytes.get());
+      }
+    }
+
+    final ValueSetIndexes valueSetIndexes = indexSupplier.as(ValueSetIndexes.class);
+    if (valueSetIndexes != null) {
+      return valueSetIndexes.forSortedValues(lazyMatchValues.get(), matchValueType);
+    }
+
+    return Filters.makePredicateIndex(
+        column,
+        selector,
+        predicateFactorySupplier.get()
+    );
+  }
+
+  @Override
+  public ValueMatcher makeMatcher(ColumnSelectorFactory factory)
+  {
+    return Filters.makeValueMatcher(factory, column, predicateFactorySupplier.get());
+  }
+
+  @Override
+  public VectorValueMatcher makeVectorMatcher(final VectorColumnSelectorFactory factory)
+  {
+    return ColumnProcessors.makeVectorProcessor(
+        column,
+        VectorValueMatcherColumnProcessorFactory.instance(),
+        factory
+    ).makeMatcher(predicateFactorySupplier.get());
+  }
+
+  @Override
+  public boolean canVectorizeMatcher(ColumnInspector inspector)
+  {
+    return true;
+  }
+
+  @Override
+  public boolean supportsRequiredColumnRewrite()
+  {
+    return true;
+  }
+
+  @Override
+  public Filter rewriteRequiredColumns(Map<String, String> columnRewrites)
+  {
+    String rewriteDimensionTo = columnRewrites.get(column);
+    if (rewriteDimensionTo == null) {
+      throw new IAE("Received a non-applicable rewrite: %s, filter's dimension: %s", columnRewrites, column);
+    }
+
+    if (rewriteDimensionTo.equals(column)) {
+      return this;
+    } else {
+      return new TypedInFilter(
+          rewriteDimensionTo,
+          matchValueType,
+          null,
+          lazyMatchValues.get(),
+          filterTuning
+      );
+    }
+  }
+
+  @Override
+  public String toString()
+  {
+    final DimFilter.DimFilterToStringBuilder builder = new DimFilter.DimFilterToStringBuilder();
+    return builder.appendDimension(column, null)
+                  .append(" IN (")
+                  .append(Joiner.on(", ").join(Iterables.transform(lazyMatchValues.get(), String::valueOf)))
+                  .append(")")
+                  .append(" (" + matchValueType + ")")
+                  .appendFilterTuning(filterTuning)
+                  .build();
+  }
+
+  @Override
+  public boolean equals(Object o)
+  {
+    if (this == o) {
+      return true;
+    }
+    if (o == null || getClass() != o.getClass()) {
+      return false;
+    }
+    TypedInFilter that = (TypedInFilter) o;
+    return column.equals(that.column) &&
+           Objects.equals(matchValueType, that.matchValueType) &&
+           compareValues(lazyMatchValues.get(), that.lazyMatchValues.get(), matchValueType) &&
+           Objects.equals(filterTuning, that.filterTuning);
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Objects.hash(lazyMatchValues.get(), column, matchValueType, filterTuning);
+  }
+
+  private byte[] computeCacheKey()
+  {
+    // Hash all values, in sorted order, as their length followed by their content.
+    final Hasher hasher = Hashing.sha256().newHasher();
+    for (Object v : lazyMatchValues.get()) {
+      if (v == null) {
+        // Encode null as length -1, no content.
+        hasher.putInt(-1);
+      } else {
+        final String s = Evals.asString(v);
+        hasher.putInt(s.length());
+        hasher.putString(s, StandardCharsets.UTF_8);
+      }
+    }
+
+    return new CacheKeyBuilder(DimFilterUtils.NEW_IN_CACHE_ID)
+        .appendString(column)
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendString(matchValueType.asTypeString())
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendByteArray(hasher.hash().asBytes())
+        .build();
+  }
+
+  private static List<?> sortValues(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Stream<?> coerced;
+    if (matchValueType.is(ValueType.LONG)) {
+      coerced = unsortedValues.stream().map(DimensionHandlerUtils::convertObjectToLong).distinct();
+    } else if (matchValueType.is(ValueType.DOUBLE)) {
+      coerced = unsortedValues.stream().map(DimensionHandlerUtils::convertObjectToDouble).distinct();
+    } else if (matchValueType.is(ValueType.FLOAT)) {
+      coerced = unsortedValues.stream().map(DimensionHandlerUtils::convertObjectToFloat).distinct();
+    } else {
+      coerced = unsortedValues.stream().distinct();
+    }
+    return coerced.sorted(matchValueType.getNullableStrategy()).collect(Collectors.toList());
+  }
+
+  /**
+   * Since jackson might translate longs into ints and such, we use the type comparator to check lists
+   * for {@link #equals(Object)} instead of directly using {@link Objects#equals(Object, Object)}
+   */
+  private static boolean compareValues(List<?> o1, List<?> o2, ColumnType matchValueType)
+  {
+    final NullableTypeStrategy comparator = matchValueType.getNullableStrategy();
+    //noinspection ObjectEquality
+    if (o1 == o2) {
+      return true;
+    }
+    if (o1 == null) {
+      return false;
+    }
+    if (o2 == null) {
+      return false;
+    }
+    final int iter = Math.min(o1.size(), o2.size());
+    for (int i = 0; i < iter; i++) {
+      final int cmp = comparator.compare(o1.get(i), o2.get(i));
+      if (cmp == 0) {
+        continue;
+      }
+      return false;
+    }
+    return o1.size() == o2.size();

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534871811


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());

Review Comment:
   changed `Utf8ValueSetIndexes` to use a List now instead of a sortedset, which required some changes in InDimFilter too but i think is ok since the valueset there also ensures its ordered when it is created



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533196315


##########
processing/src/test/java/org/apache/druid/segment/filter/TypedInFilterTests.java:
##########
@@ -0,0 +1,509 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.filter;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Function;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.RangeSet;
+import nl.jqno.equalsverifier.EqualsVerifier;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.query.filter.DimFilter;
+import org.apache.druid.query.filter.Filter;
+import org.apache.druid.query.filter.NotDimFilter;
+import org.apache.druid.query.filter.TypedInFilter;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.StorageAdapter;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.Closeable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+@RunWith(Enclosed.class)
+public class TypedInFilterTests

Review Comment:
   hmm, the new filter doesn't actually support default value mode though, so need to think of a way to handle that (as well as extractionFn tests). the other thing i could think of that might be a problem is that converting doubles to strings is sometimes ugly, so might still need to override some of those tests anyway depending on the mode
   
   in the worst case i'll add javadoc comment to that test, and other classic filter tests that anything added there should be considered to also be added to the strongly typed replacement/equivalents tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534874130


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      this.unsortedValues = values;
+      this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());
+        for (Object s : lazyMatchValues.get()) {
+          matchValueBytes.add(StringUtils.toUtf8ByteBuffer(Evals.asString(s)));
+        }
+        return matchValueBytes;
+      });
+    } else {
+      this.lazyMatchValueBytes = null;
+    }
+
+    this.predicateFactorySupplier = Suppliers.memoize(
+        () -> new InFilterDruidPredicateFactory(lazyMatchValues.get(), matchValueType)
+    );
+    this.cacheKeySupplier = Suppliers.memoize(this::computeCacheKey);
+  }
+
+  @JsonProperty
+  public String getColumn()
+  {
+    return column;
+  }
+
+  @JsonProperty
+  public List<?> getSortedValues()
+  {
+    return lazyMatchValues.get();
+  }
+
+  @JsonProperty
+  public ColumnType getMatchValueType()
+  {
+    return matchValueType;
+  }
+
+  @Nullable
+  @JsonInclude(JsonInclude.Include.NON_NULL)
+  @JsonProperty
+  public FilterTuning getFilterTuning()
+  {
+    return filterTuning;
+  }
+
+  @Override
+  public byte[] getCacheKey()
+  {
+    return cacheKeySupplier.get();
+  }
+
+  @Override
+  public DimFilter optimize(final boolean mayIncludeUnknown)
+  {
+    final List<?> matchValues = lazyMatchValues.get();
+    if (matchValues.isEmpty()) {
+      return FalseDimFilter.instance();
+    } else if (matchValues.size() == 1) {
+      if (matchValues.get(0) == null) {
+        return NullFilter.forColumn(column);
+      }
+      return new EqualityFilter(
+          column,
+          matchValueType,
+          matchValues.iterator().next(),
+          filterTuning
+      );
+    }
+    return this;
+  }
+
+  @Override
+  public Filter toFilter()
+  {
+    return this;
+  }
+
+  @Nullable
+  @Override
+  public RangeSet<String> getDimensionRangeSet(String dimension)
+  {
+    if (!Objects.equals(getColumn(), dimension)) {
+      return null;
+    }
+    RangeSet<String> retSet = TreeRangeSet.create();
+    for (Object value : lazyMatchValues.get()) {
+      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534873763


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;

Review Comment:
   added javadocs to clarify this hopefully



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534872215


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter

Review Comment:
   added, also added recommendations to InDimFilter, SelectorDimFilter, and BoundDimFilter to use TypedInFilter, EqualityFilter, NullFilter, and RangeFilter instead



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534873957


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.

Review Comment:
   added



##########
processing/src/main/java/org/apache/druid/query/filter/DimFilterUtils.java:
##########
@@ -62,6 +62,7 @@ public class DimFilterUtils
   static final byte RANGE_CACHE_ID = 0x14;
   static final byte IS_FILTER_BOOLEAN_FILTER_CACHE_ID = 0x15;
   static final byte ARRAY_CONTAINS_CACHE_ID = 0x16;
+  static final byte NEW_IN_CACHE_ID = 0x17;

Review Comment:
   done



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {

Review Comment:
   added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533197382


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      this.unsortedValues = values;
+      this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());
+        for (Object s : lazyMatchValues.get()) {
+          matchValueBytes.add(StringUtils.toUtf8ByteBuffer(Evals.asString(s)));
+        }
+        return matchValueBytes;
+      });
+    } else {
+      this.lazyMatchValueBytes = null;
+    }
+
+    this.predicateFactorySupplier = Suppliers.memoize(
+        () -> new InFilterDruidPredicateFactory(lazyMatchValues.get(), matchValueType)
+    );
+    this.cacheKeySupplier = Suppliers.memoize(this::computeCacheKey);
+  }
+
+  @JsonProperty
+  public String getColumn()
+  {
+    return column;
+  }
+
+  @JsonProperty
+  public List<?> getSortedValues()
+  {
+    return lazyMatchValues.get();
+  }
+
+  @JsonProperty
+  public ColumnType getMatchValueType()
+  {
+    return matchValueType;
+  }
+
+  @Nullable
+  @JsonInclude(JsonInclude.Include.NON_NULL)
+  @JsonProperty
+  public FilterTuning getFilterTuning()
+  {
+    return filterTuning;
+  }
+
+  @Override
+  public byte[] getCacheKey()
+  {
+    return cacheKeySupplier.get();
+  }
+
+  @Override
+  public DimFilter optimize(final boolean mayIncludeUnknown)
+  {
+    final List<?> matchValues = lazyMatchValues.get();
+    if (matchValues.isEmpty()) {
+      return FalseDimFilter.instance();
+    } else if (matchValues.size() == 1) {
+      if (matchValues.get(0) == null) {
+        return NullFilter.forColumn(column);
+      }
+      return new EqualityFilter(
+          column,
+          matchValueType,
+          matchValues.iterator().next(),
+          filterTuning
+      );
+    }
+    return this;
+  }
+
+  @Override
+  public Filter toFilter()
+  {
+    return this;
+  }
+
+  @Nullable
+  @Override
+  public RangeSet<String> getDimensionRangeSet(String dimension)
+  {
+    if (!Objects.equals(getColumn(), dimension)) {
+      return null;
+    }
+    RangeSet<String> retSet = TreeRangeSet.create();
+    for (Object value : lazyMatchValues.get()) {
+      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));

Review Comment:
   stale code from when i was originally aspiring to support both modes, will remove



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533189728


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {

Review Comment:
   oh yea, good point



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534989906


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {

Review Comment:
   Hmm, dedupe after quicksort is O(n); probably wouldn't notice it much next to the quicksort. It might even be faster to do quicksort + dedupe pass vs. doing tree-sort. Although, this probably doesn't matter _that_ much, since it's only happening once per query. The sorted set approach is IMO OK for this patch, esp. since that's the same thing the `in` filter is doing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533189333


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(

Review Comment:
   ah yea, that makes sense, will change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533188709


##########
processing/src/main/java/org/apache/druid/segment/nested/ScalarDoubleColumnAndIndexSupplier.java:
##########
@@ -258,6 +265,70 @@ public <T> T computeBitmapResult(BitmapResultFactory<T> bitmapResultFactory, boo
     }
   }
 
+  private final class DoubleValueSetIndexes implements ValueSetIndexes
+  {
+    final FixedIndexed<Double> dictionary = doubleDictionarySupplier.get();
+
+    @Nullable
+    @Override
+    public BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType)
+    {
+      final boolean matchNull = sortedValues.get(0) == null;
+      final Supplier<ImmutableBitmap> unknownsIndex = () -> {
+        if (!matchNull && dictionary.get(0) == null) {
+          return valueIndexes.get(0);
+        }
+        return null;
+      };
+
+      // values are doubles and ordered in double order
+      if (matchValueType.is(ValueType.DOUBLE)) {

Review Comment:
   yea probably, also for the other index implementations.. i can look into it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533198022


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {

Review Comment:
   im against it, it defeats the purpose of keeping them split, which is to not do extra work unless we need to. if people want to manually craft json that is broken then that's on them. if we document this filter i'll be sure to make it super clear that if this is set that it must be sorted exactly the same as the matchValueType, or maybe just not document that this property exists at all 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533179974


##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/ArrayOverlapOperatorConversion.java:
##########
@@ -127,7 +130,7 @@ public DimFilter toDruidFilter(
         // to create an empty array with no argument, we just return null.
         return null;
       } else if (arrayElements.length == 1) {
-        if (plannerContext.isUseBoundsAndSelectors()) {
+        if (plannerContext.isUseBoundsAndSelectors() || (!simpleExtractionExpr.isDirectColumnAccess() && virtualColumnRegistry == null)) {

Review Comment:
   it looks like it can be null in some join cases, https://github.com/apache/druid/blob/master/sql/src/main/java/org/apache/druid/sql/calcite/rel/DruidJoinQueryRel.java#L497 calls it with null. will add a guard



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533183035


##########
processing/src/main/java/org/apache/druid/segment/index/semantic/ValueSetIndexes.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.index.semantic;
+
+import com.google.common.base.Supplier;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.collections.bitmap.BitmapFactory;
+import org.apache.druid.collections.bitmap.ImmutableBitmap;
+import org.apache.druid.segment.column.TypeSignature;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.SimpleImmutableBitmapDelegatingIterableIndex;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+
+public interface ValueSetIndexes
+{
+  double SORTED_MERGE_RATIO_THRESHOLD = 0.12D;
+  int SIZE_WORTH_CHECKING_MIN = 8;
+
+  /**
+   * Get the wrapped {@link ImmutableBitmap} corresponding to the specified set of values (if they are contained in the
+   * underlying column). The set must be sorted using the comparator of the supplied matchValueType.
+   *
+   * @param sortedValues   values to match, sorted in matchValueType order
+   * @param matchValueType type of the value to match, used to assist conversion from the match value type to the column
+   *                       value type
+   * @return               {@link ImmutableBitmap} corresponding to the rows which match the values, or null if an index
+   *                       connot be computed for the supplied value type
+   */
+  @Nullable
+  BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType);
+
+  static <T> BitmapColumnIndex getIndexFromSortedIteratorSortedMerged(

Review Comment:
   there are 3 total methods, the sorted merge (`getIndexFromSortedIteratorSortedMerged`), the binary search with short-circuit when the values are sorted the same way as the dictionary and we get pas the end of the dictionary (`getIndexFromSortedIterator`), and the binary search when the values are not necessarily sorted the same way at the dictionary and so we must iterate them all (`getIndexFromIterator`).
   
   Will add javadocs and try to give better names



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534871090


##########
processing/src/test/java/org/apache/druid/segment/filter/TypedInFilterTests.java:
##########
@@ -0,0 +1,509 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.filter;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Function;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.RangeSet;
+import nl.jqno.equalsverifier.EqualsVerifier;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.DruidException;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.query.filter.DimFilter;
+import org.apache.druid.query.filter.Filter;
+import org.apache.druid.query.filter.NotDimFilter;
+import org.apache.druid.query.filter.TypedInFilter;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.StorageAdapter;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.testing.InitializedNullHandlingTest;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Assume;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.Closeable;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+@RunWith(Enclosed.class)
+public class TypedInFilterTests

Review Comment:
   ok, consolidated these tests. its kind of ugly, but i think ok



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());
+        for (Object s : lazyMatchValues.get()) {
+          matchValueBytes.add(StringUtils.toUtf8ByteBuffer(Evals.asString(s)));
+        }
+        return matchValueBytes;
+      });
+    } else {
+      this.lazyMatchValueBytes = null;
+    }
+
+    this.predicateFactorySupplier = Suppliers.memoize(
+        () -> new PredicateFactory(lazyMatchValues.get(), matchValueType)
+    );
+    this.cacheKeySupplier = Suppliers.memoize(this::computeCacheKey);
+  }
+
+  @JsonProperty
+  public String getColumn()
+  {
+    return column;
+  }
+
+  @JsonProperty
+  public List<?> getSortedValues()
+  {
+    return lazyMatchValues.get();
+  }
+
+  @JsonProperty
+  public ColumnType getMatchValueType()
+  {
+    return matchValueType;
+  }
+
+  @Nullable
+  @JsonInclude(JsonInclude.Include.NON_NULL)
+  @JsonProperty
+  public FilterTuning getFilterTuning()
+  {
+    return filterTuning;
+  }
+
+  @Override
+  public byte[] getCacheKey()
+  {
+    return cacheKeySupplier.get();
+  }
+
+  @Override
+  public DimFilter optimize(final boolean mayIncludeUnknown)
+  {
+    final List<?> matchValues = lazyMatchValues.get();
+    if (matchValues.isEmpty()) {
+      return FalseDimFilter.instance();
+    } else if (matchValues.size() == 1) {
+      if (matchValues.get(0) == null) {
+        return NullFilter.forColumn(column);
+      }
+      return new EqualityFilter(
+          column,
+          matchValueType,
+          matchValues.iterator().next(),
+          filterTuning
+      );
+    }
+    return this;
+  }
+
+  @Override
+  public Filter toFilter()
+  {
+    return this;
+  }
+
+  @Nullable
+  @Override
+  public RangeSet<String> getDimensionRangeSet(String dimension)
+  {
+    if (!Objects.equals(getColumn(), dimension)) {
+      return null;
+    }
+    RangeSet<String> retSet = TreeRangeSet.create();
+    for (Object value : lazyMatchValues.get()) {
+      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));
+      if (valueEquivalent == null) {
+        // Case when SQL compatible null handling is enabled
+        // Range.singleton(null) is invalid, so use the fact that
+        // only null values are less than empty string.
+        retSet.add(Range.lessThan(""));
+      } else {
+        retSet.add(Range.singleton(valueEquivalent));
+      }
+    }
+    return retSet;
+  }
+
+  @Override
+  public Set<String> getRequiredColumns()
+  {
+    return ImmutableSet.of(column);
+  }
+
+  @Override
+  @Nullable
+  public BitmapColumnIndex getBitmapColumnIndex(ColumnIndexSelector selector)
+  {
+    if (!Filters.checkFilterTuningUseIndex(column, selector, filterTuning)) {
+      return null;
+    }
+    final ColumnIndexSupplier indexSupplier = selector.getIndexSupplier(column);
+
+    if (indexSupplier == null) {
+      // column doesn't exist, match against null
+      DruidPredicateMatch match = predicateFactorySupplier.get().makeStringPredicate().apply(null);
+      return Filters.makeMissingColumnNullIndex(match, selector);
+    }
+
+    if (lazyMatchValueBytes != null) {
+      final Utf8ValueSetIndexes utf8ValueSetIndexes = indexSupplier.as(Utf8ValueSetIndexes.class);
+      if (utf8ValueSetIndexes != null) {
+        return utf8ValueSetIndexes.forSortedValuesUtf8(lazyMatchValueBytes.get());
+      }
+    }
+
+    final ValueSetIndexes valueSetIndexes = indexSupplier.as(ValueSetIndexes.class);
+    if (valueSetIndexes != null) {
+      return valueSetIndexes.forSortedValues(lazyMatchValues.get(), matchValueType);
+    }
+
+    return Filters.makePredicateIndex(
+        column,
+        selector,
+        predicateFactorySupplier.get()
+    );
+  }
+
+  @Override
+  public ValueMatcher makeMatcher(ColumnSelectorFactory factory)
+  {
+    return Filters.makeValueMatcher(factory, column, predicateFactorySupplier.get());
+  }
+
+  @Override
+  public VectorValueMatcher makeVectorMatcher(final VectorColumnSelectorFactory factory)
+  {
+    return ColumnProcessors.makeVectorProcessor(
+        column,
+        VectorValueMatcherColumnProcessorFactory.instance(),
+        factory
+    ).makeMatcher(predicateFactorySupplier.get());
+  }
+
+  @Override
+  public boolean canVectorizeMatcher(ColumnInspector inspector)
+  {
+    return true;
+  }
+
+  @Override
+  public boolean supportsRequiredColumnRewrite()
+  {
+    return true;
+  }
+
+  @Override
+  public Filter rewriteRequiredColumns(Map<String, String> columnRewrites)
+  {
+    String rewriteDimensionTo = columnRewrites.get(column);
+    if (rewriteDimensionTo == null) {
+      throw new IAE("Received a non-applicable rewrite: %s, filter's dimension: %s", columnRewrites, column);
+    }
+
+    if (rewriteDimensionTo.equals(column)) {
+      return this;
+    } else {
+      return new TypedInFilter(
+          rewriteDimensionTo,
+          matchValueType,
+          null,
+          lazyMatchValues.get(),
+          filterTuning
+      );
+    }
+  }
+
+  @Override
+  public String toString()
+  {
+    final DimFilter.DimFilterToStringBuilder builder = new DimFilter.DimFilterToStringBuilder();
+    return builder.appendDimension(column, null)
+                  .append(" IN (")
+                  .append(Joiner.on(", ").join(Iterables.transform(lazyMatchValues.get(), String::valueOf)))
+                  .append(")")
+                  .append(" (" + matchValueType + ")")
+                  .appendFilterTuning(filterTuning)
+                  .build();
+  }
+
+  @Override
+  public boolean equals(Object o)
+  {
+    if (this == o) {
+      return true;
+    }
+    if (o == null || getClass() != o.getClass()) {
+      return false;
+    }
+    TypedInFilter that = (TypedInFilter) o;
+    return column.equals(that.column) &&
+           Objects.equals(matchValueType, that.matchValueType) &&
+           compareValues(lazyMatchValues.get(), that.lazyMatchValues.get(), matchValueType) &&
+           Objects.equals(filterTuning, that.filterTuning);
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Objects.hash(lazyMatchValues.get(), column, matchValueType, filterTuning);
+  }
+
+  private byte[] computeCacheKey()
+  {
+    // Hash all values, in sorted order, as their length followed by their content.
+    final Hasher hasher = Hashing.sha256().newHasher();
+    for (Object v : lazyMatchValues.get()) {
+      if (v == null) {
+        // Encode null as length -1, no content.
+        hasher.putInt(-1);
+      } else {
+        final String s = Evals.asString(v);
+        hasher.putInt(s.length());
+        hasher.putString(s, StandardCharsets.UTF_8);
+      }
+    }
+
+    return new CacheKeyBuilder(DimFilterUtils.NEW_IN_CACHE_ID)
+        .appendString(column)
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendString(matchValueType.asTypeString())
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendByteArray(hasher.hash().asBytes())
+        .build();
+  }
+
+  private static boolean checkSorted(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Comparator<Object> comparator = matchValueType.getNullableStrategy();
+    Object prev = null;
+    boolean needsCoerceCheck = true;
+    for (Object o : unsortedValues) {
+      if (needsCoerceCheck && o != null) {
+        Object coerced = coerceValue(o, matchValueType);
+        //noinspection ObjectEquality
+        if (coerced != o) {
+          return false;
+        }
+        needsCoerceCheck = false;
+      }
+      if (prev != null && comparator.compare(prev, o) >= 0) {
+        return false;
+      }
+      prev = o;
+    }
+    return true;
+  }
+
+  @Nullable
+  private static Object coerceValue(@Nullable Object o, ColumnType matchValueType)
+  {
+    if (o == null) {
+      return o;
+    }
+    switch (matchValueType.getType()) {
+      case STRING:
+        return DimensionHandlerUtils.convertObjectToString(o);
+      case LONG:
+        return DimensionHandlerUtils.convertObjectToLong(o);
+      case FLOAT:
+        return DimensionHandlerUtils.convertObjectToFloat(o);
+      case DOUBLE:
+        return DimensionHandlerUtils.convertObjectToDouble(o);
+      default:
+        throw InvalidInput.exception("Unsupported matchValueType[%s]", matchValueType);
+    }
+  }
+
+  private static List<?> sortValues(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Object[] array = unsortedValues.toArray(new Object[0]);
+    // check if values need coerced
+    for (int i = 0; i < array.length; i++) {
+      Object coerced = coerceValue(array[i], matchValueType);
+      //noinspection ObjectEquality
+      if (coerced != null && array[i] == coerced) {
+        // assume list is all same type objects...
+        break;
+      }
+      array[i] = coerced;
+    }
+    ObjectArrays.quickSort(array, matchValueType.getNullableStrategy());
+    return Arrays.asList(array);
+  }
+
+  /**
+   * Since jackson might translate longs into ints and such, we use the type comparator to check lists
+   * for {@link #equals(Object)} instead of directly using {@link Objects#equals(Object, Object)}
+   */
+  private static boolean compareValues(List<?> o1, List<?> o2, ColumnType matchValueType)
+  {
+    final NullableTypeStrategy comparator = matchValueType.getNullableStrategy();
+    //noinspection ObjectEquality
+    if (o1 == o2) {
+      return true;
+    }
+    if (o1 == null) {
+      return false;
+    }
+    if (o2 == null) {
+      return false;
+    }
+    final int iter = Math.min(o1.size(), o2.size());
+    for (int i = 0; i < iter; i++) {
+      final int cmp = comparator.compare(o1.get(i), o2.get(i));
+      if (cmp == 0) {
+        continue;
+      }
+      return false;
+    }
+    return o1.size() == o2.size();
+  }
+
+  private static DruidObjectPredicate<String> createStringPredicate(
+      final List sortedValues,

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534871389


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {

Review Comment:
   i went back to using a sorted set to order and dedupe since I could think of a very efficient way to do this otherwise other than creating my own modified version of quickSort that also dedupes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534873016


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(

Review Comment:
   moved this check to toFilter and optimize methods (the latter since it might rewrite to an invalid equality filter in default value mode if the match set is a single value with the empty string, which is translated into null in default value mode which should use a null filter instead of equality filter)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis merged PR #16039:
URL: https://github.com/apache/druid/pull/16039


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1515960661


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"

Review Comment:
   ```suggestion
             "No performance gains for you if you continue to use non-SQL compatible mode"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533193264


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());
+        for (Object s : lazyMatchValues.get()) {
+          matchValueBytes.add(StringUtils.toUtf8ByteBuffer(Evals.asString(s)));
+        }
+        return matchValueBytes;
+      });
+    } else {
+      this.lazyMatchValueBytes = null;
+    }
+
+    this.predicateFactorySupplier = Suppliers.memoize(
+        () -> new PredicateFactory(lazyMatchValues.get(), matchValueType)
+    );
+    this.cacheKeySupplier = Suppliers.memoize(this::computeCacheKey);
+  }
+
+  @JsonProperty
+  public String getColumn()
+  {
+    return column;
+  }
+
+  @JsonProperty
+  public List<?> getSortedValues()
+  {
+    return lazyMatchValues.get();
+  }
+
+  @JsonProperty
+  public ColumnType getMatchValueType()
+  {
+    return matchValueType;
+  }
+
+  @Nullable
+  @JsonInclude(JsonInclude.Include.NON_NULL)
+  @JsonProperty
+  public FilterTuning getFilterTuning()
+  {
+    return filterTuning;
+  }
+
+  @Override
+  public byte[] getCacheKey()
+  {
+    return cacheKeySupplier.get();
+  }
+
+  @Override
+  public DimFilter optimize(final boolean mayIncludeUnknown)
+  {
+    final List<?> matchValues = lazyMatchValues.get();
+    if (matchValues.isEmpty()) {
+      return FalseDimFilter.instance();
+    } else if (matchValues.size() == 1) {
+      if (matchValues.get(0) == null) {
+        return NullFilter.forColumn(column);
+      }
+      return new EqualityFilter(
+          column,
+          matchValueType,
+          matchValues.iterator().next(),
+          filterTuning
+      );
+    }
+    return this;
+  }
+
+  @Override
+  public Filter toFilter()
+  {
+    return this;
+  }
+
+  @Nullable
+  @Override
+  public RangeSet<String> getDimensionRangeSet(String dimension)
+  {
+    if (!Objects.equals(getColumn(), dimension)) {
+      return null;
+    }
+    RangeSet<String> retSet = TreeRangeSet.create();
+    for (Object value : lazyMatchValues.get()) {
+      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));
+      if (valueEquivalent == null) {
+        // Case when SQL compatible null handling is enabled
+        // Range.singleton(null) is invalid, so use the fact that
+        // only null values are less than empty string.
+        retSet.add(Range.lessThan(""));
+      } else {
+        retSet.add(Range.singleton(valueEquivalent));
+      }
+    }
+    return retSet;
+  }
+
+  @Override
+  public Set<String> getRequiredColumns()
+  {
+    return ImmutableSet.of(column);
+  }
+
+  @Override
+  @Nullable
+  public BitmapColumnIndex getBitmapColumnIndex(ColumnIndexSelector selector)
+  {
+    if (!Filters.checkFilterTuningUseIndex(column, selector, filterTuning)) {
+      return null;
+    }
+    final ColumnIndexSupplier indexSupplier = selector.getIndexSupplier(column);
+
+    if (indexSupplier == null) {
+      // column doesn't exist, match against null
+      DruidPredicateMatch match = predicateFactorySupplier.get().makeStringPredicate().apply(null);
+      return Filters.makeMissingColumnNullIndex(match, selector);
+    }
+
+    if (lazyMatchValueBytes != null) {
+      final Utf8ValueSetIndexes utf8ValueSetIndexes = indexSupplier.as(Utf8ValueSetIndexes.class);
+      if (utf8ValueSetIndexes != null) {
+        return utf8ValueSetIndexes.forSortedValuesUtf8(lazyMatchValueBytes.get());
+      }
+    }
+
+    final ValueSetIndexes valueSetIndexes = indexSupplier.as(ValueSetIndexes.class);
+    if (valueSetIndexes != null) {
+      return valueSetIndexes.forSortedValues(lazyMatchValues.get(), matchValueType);
+    }
+
+    return Filters.makePredicateIndex(
+        column,
+        selector,
+        predicateFactorySupplier.get()
+    );
+  }
+
+  @Override
+  public ValueMatcher makeMatcher(ColumnSelectorFactory factory)
+  {
+    return Filters.makeValueMatcher(factory, column, predicateFactorySupplier.get());
+  }
+
+  @Override
+  public VectorValueMatcher makeVectorMatcher(final VectorColumnSelectorFactory factory)
+  {
+    return ColumnProcessors.makeVectorProcessor(
+        column,
+        VectorValueMatcherColumnProcessorFactory.instance(),
+        factory
+    ).makeMatcher(predicateFactorySupplier.get());
+  }
+
+  @Override
+  public boolean canVectorizeMatcher(ColumnInspector inspector)
+  {
+    return true;
+  }
+
+  @Override
+  public boolean supportsRequiredColumnRewrite()
+  {
+    return true;
+  }
+
+  @Override
+  public Filter rewriteRequiredColumns(Map<String, String> columnRewrites)
+  {
+    String rewriteDimensionTo = columnRewrites.get(column);
+    if (rewriteDimensionTo == null) {
+      throw new IAE("Received a non-applicable rewrite: %s, filter's dimension: %s", columnRewrites, column);
+    }
+
+    if (rewriteDimensionTo.equals(column)) {
+      return this;
+    } else {
+      return new TypedInFilter(
+          rewriteDimensionTo,
+          matchValueType,
+          null,
+          lazyMatchValues.get(),
+          filterTuning
+      );
+    }
+  }
+
+  @Override
+  public String toString()
+  {
+    final DimFilter.DimFilterToStringBuilder builder = new DimFilter.DimFilterToStringBuilder();
+    return builder.appendDimension(column, null)
+                  .append(" IN (")
+                  .append(Joiner.on(", ").join(Iterables.transform(lazyMatchValues.get(), String::valueOf)))
+                  .append(")")
+                  .append(" (" + matchValueType + ")")
+                  .appendFilterTuning(filterTuning)
+                  .build();
+  }
+
+  @Override
+  public boolean equals(Object o)
+  {
+    if (this == o) {
+      return true;
+    }
+    if (o == null || getClass() != o.getClass()) {
+      return false;
+    }
+    TypedInFilter that = (TypedInFilter) o;
+    return column.equals(that.column) &&
+           Objects.equals(matchValueType, that.matchValueType) &&
+           compareValues(lazyMatchValues.get(), that.lazyMatchValues.get(), matchValueType) &&
+           Objects.equals(filterTuning, that.filterTuning);
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Objects.hash(lazyMatchValues.get(), column, matchValueType, filterTuning);
+  }
+
+  private byte[] computeCacheKey()
+  {
+    // Hash all values, in sorted order, as their length followed by their content.
+    final Hasher hasher = Hashing.sha256().newHasher();
+    for (Object v : lazyMatchValues.get()) {
+      if (v == null) {
+        // Encode null as length -1, no content.
+        hasher.putInt(-1);
+      } else {
+        final String s = Evals.asString(v);
+        hasher.putInt(s.length());
+        hasher.putString(s, StandardCharsets.UTF_8);
+      }
+    }
+
+    return new CacheKeyBuilder(DimFilterUtils.NEW_IN_CACHE_ID)
+        .appendString(column)
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendString(matchValueType.asTypeString())
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendByteArray(hasher.hash().asBytes())
+        .build();
+  }
+
+  private static boolean checkSorted(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Comparator<Object> comparator = matchValueType.getNullableStrategy();
+    Object prev = null;
+    boolean needsCoerceCheck = true;
+    for (Object o : unsortedValues) {
+      if (needsCoerceCheck && o != null) {
+        Object coerced = coerceValue(o, matchValueType);
+        //noinspection ObjectEquality
+        if (coerced != o) {
+          return false;
+        }
+        needsCoerceCheck = false;
+      }
+      if (prev != null && comparator.compare(prev, o) >= 0) {
+        return false;
+      }
+      prev = o;
+    }
+    return true;
+  }
+
+  @Nullable
+  private static Object coerceValue(@Nullable Object o, ColumnType matchValueType)
+  {
+    if (o == null) {
+      return o;
+    }
+    switch (matchValueType.getType()) {
+      case STRING:
+        return DimensionHandlerUtils.convertObjectToString(o);
+      case LONG:
+        return DimensionHandlerUtils.convertObjectToLong(o);
+      case FLOAT:
+        return DimensionHandlerUtils.convertObjectToFloat(o);
+      case DOUBLE:
+        return DimensionHandlerUtils.convertObjectToDouble(o);
+      default:
+        throw InvalidInput.exception("Unsupported matchValueType[%s]", matchValueType);
+    }
+  }
+
+  private static List<?> sortValues(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Object[] array = unsortedValues.toArray(new Object[0]);
+    // check if values need coerced
+    for (int i = 0; i < array.length; i++) {
+      Object coerced = coerceValue(array[i], matchValueType);
+      //noinspection ObjectEquality
+      if (coerced != null && array[i] == coerced) {
+        // assume list is all same type objects...

Review Comment:
   hmm, yea I guess i could always coerce while sorting just to be safe, was trying to short-circuit if it looked like it wasnt going to be necessary and was just assuming people wouldn't troll the set with mixed type junk like strings and numbers over json (and a mix of like ints and longs wouldn't really be a problem because the column type comparators handle the stuff as numbers rather than exact types to guard against json serde shenanigans)
   
   i guess i was still on the fence of whether or not we document. we did document the other new native typed filters, so was sort of assuming we would this one too, especially since there is a performance improvement



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533197171


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {

Review Comment:
   yea, might as well with the way things are now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533200679


##########
processing/src/main/java/org/apache/druid/segment/nested/ScalarDoubleColumnAndIndexSupplier.java:
##########
@@ -258,6 +265,70 @@ public <T> T computeBitmapResult(BitmapResultFactory<T> bitmapResultFactory, boo
     }
   }
 
+  private final class DoubleValueSetIndexes implements ValueSetIndexes
+  {
+    final FixedIndexed<Double> dictionary = doubleDictionarySupplier.get();
+
+    @Nullable
+    @Override
+    public BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType)
+    {
+      final boolean matchNull = sortedValues.get(0) == null;
+      final Supplier<ImmutableBitmap> unknownsIndex = () -> {
+        if (!matchNull && dictionary.get(0) == null) {
+          return valueIndexes.get(0);
+        }
+        return null;
+      };
+
+      // values are doubles and ordered in double order
+      if (matchValueType.is(ValueType.DOUBLE)) {

Review Comment:
   i guess the downside to doing this is that i was hoping at some point to specialize the value dictionaries for primitive values (e.g. #12846) and combining them would make that not very easy since would need a generic parameter instead of java primitive. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534864286


##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/ArrayOverlapOperatorConversion.java:
##########
@@ -127,7 +130,7 @@ public DimFilter toDruidFilter(
         // to create an empty array with no argument, we just return null.
         return null;
       } else if (arrayElements.length == 1) {
-        if (plannerContext.isUseBoundsAndSelectors()) {
+        if (plannerContext.isUseBoundsAndSelectors() || (!simpleExtractionExpr.isDirectColumnAccess() && virtualColumnRegistry == null)) {

Review Comment:
   ok, reworked this a bit, if the virtual column registry is null it seems equivalent to just use an expression filter rather than forcing a selector filter + extractionFn, so I did that for now, since it seems better to do that and get away from reliance on using extractionFn defined on filters



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534990675


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {

Review Comment:
   probably best to document it, since it will show up in `explain plan` from SQL, and people will wonder what it means.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533190347


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());

Review Comment:
   yea, i would need to change the interface of `Utf8ValueSetIndexes` to not require a sorted set, but can go ahead and do that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1531475670


##########
processing/src/main/java/org/apache/druid/segment/nested/ScalarDoubleColumnAndIndexSupplier.java:
##########
@@ -258,6 +265,70 @@ public <T> T computeBitmapResult(BitmapResultFactory<T> bitmapResultFactory, boo
     }
   }
 
+  private final class DoubleValueSetIndexes implements ValueSetIndexes
+  {
+    final FixedIndexed<Double> dictionary = doubleDictionarySupplier.get();
+
+    @Nullable
+    @Override
+    public BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType)
+    {
+      final boolean matchNull = sortedValues.get(0) == null;
+      final Supplier<ImmutableBitmap> unknownsIndex = () -> {
+        if (!matchNull && dictionary.get(0) == null) {
+          return valueIndexes.get(0);
+        }
+        return null;
+      };
+
+      // values are doubles and ordered in double order
+      if (matchValueType.is(ValueType.DOUBLE)) {

Review Comment:
   Is it possible to minimize the copied code between here and the `LONG` version, such as by using a shared helper somewhere?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533181036


##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/ArrayOverlapOperatorConversion.java:
##########
@@ -127,7 +130,7 @@ public DimFilter toDruidFilter(
         // to create an empty array with no argument, we just return null.
         return null;
       } else if (arrayElements.length == 1) {
-        if (plannerContext.isUseBoundsAndSelectors()) {
+        if (plannerContext.isUseBoundsAndSelectors() || (!simpleExtractionExpr.isDirectColumnAccess() && virtualColumnRegistry == null)) {

Review Comment:
   oh wait, the else branch here only uses virtual column registry if its not a direct column access, so the registry cannot be null or else it would have made a selector filter with an extractionFn



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533198752


##########
processing/src/main/java/org/apache/druid/segment/index/semantic/ValueSetIndexes.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.index.semantic;
+
+import com.google.common.base.Supplier;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.collections.bitmap.BitmapFactory;
+import org.apache.druid.collections.bitmap.ImmutableBitmap;
+import org.apache.druid.segment.column.TypeSignature;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.SimpleImmutableBitmapDelegatingIterableIndex;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+
+public interface ValueSetIndexes

Review Comment:
   the methods that callers use have javadocs which is consistent with all of these other interfaces.. but will maybe fill them in with some generic bit about getting `BitmapColumnIndex` for sets of values or .. something along with other interfaces



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1535392032


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {

Review Comment:
   ok, so got around to measuring it and went back to quicksort, even with coercion and dedupe is still better on bigger sets 
   
   ```
   set:
   Benchmark                           (filterSize)  Mode  Cnt     Score     Error  Units
   InFilterBenchmark.sortFilterValues             1  avgt    3     0.021 ±   0.006  us/op
   InFilterBenchmark.sortFilterValues            10  avgt    3     0.323 ±   0.035  us/op
   InFilterBenchmark.sortFilterValues           100  avgt    3     4.521 ±   0.483  us/op
   InFilterBenchmark.sortFilterValues          1000  avgt    3   104.984 ±  11.389  us/op
   InFilterBenchmark.sortFilterValues         10000  avgt    3  1906.647 ± 922.912  us/op
   
   quicksort:
   Benchmark                           (filterSize)  Mode  Cnt     Score     Error  Units
   InFilterBenchmark.sortFilterValues             1  avgt    3     0.021 ±   0.005  us/op
   InFilterBenchmark.sortFilterValues            10  avgt    3     0.574 ±   0.083  us/op
   InFilterBenchmark.sortFilterValues           100  avgt    3     4.678 ±   1.100  us/op
   InFilterBenchmark.sortFilterValues          1000  avgt    3    84.306 ±  10.615  us/op
   InFilterBenchmark.sortFilterValues         10000  avgt    3  1555.324 ± 228.895  us/op
   ```
   (I didn't add this benchmark to the PR)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534509952


##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/ArrayOverlapOperatorConversion.java:
##########
@@ -127,7 +130,7 @@ public DimFilter toDruidFilter(
         // to create an empty array with no argument, we just return null.
         return null;
       } else if (arrayElements.length == 1) {
-        if (plannerContext.isUseBoundsAndSelectors()) {
+        if (plannerContext.isUseBoundsAndSelectors() || (!simpleExtractionExpr.isDirectColumnAccess() && virtualColumnRegistry == null)) {

Review Comment:
   👍  in that case a comment would be useful? it seemed to confuse both us of and CodeQL



##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/ArrayOverlapOperatorConversion.java:
##########
@@ -127,7 +130,7 @@ public DimFilter toDruidFilter(
         // to create an empty array with no argument, we just return null.
         return null;
       } else if (arrayElements.length == 1) {
-        if (plannerContext.isUseBoundsAndSelectors()) {
+        if (plannerContext.isUseBoundsAndSelectors() || (!simpleExtractionExpr.isDirectColumnAccess() && virtualColumnRegistry == null)) {

Review Comment:
   👍  in that case a comment would be useful? it seemed to confuse both u and CodeQL



##########
sql/src/main/java/org/apache/druid/sql/calcite/expression/builtin/ArrayOverlapOperatorConversion.java:
##########
@@ -127,7 +130,7 @@ public DimFilter toDruidFilter(
         // to create an empty array with no argument, we just return null.
         return null;
       } else if (arrayElements.length == 1) {
-        if (plannerContext.isUseBoundsAndSelectors()) {
+        if (plannerContext.isUseBoundsAndSelectors() || (!simpleExtractionExpr.isDirectColumnAccess() && virtualColumnRegistry == null)) {

Review Comment:
   👍  in that case a comment would be useful? it seemed to confuse both us of and CodeQL



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1512205368


##########
extensions-core/druid-ranger-security/pom.xml:
##########
@@ -45,6 +45,7 @@
                 <groupId>com.amazonaws</groupId>
                 <artifactId>aws-java-sdk-bundle</artifactId>
                 <version>${aws.sdk.version}</version>
+                <scope>provided</scope>

Review Comment:
   why this change? 



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"

Review Comment:
   ```suggestion
             "No performance gains for you if you continue to use non-SQL compatible mode"
   ```



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.

Review Comment:
   Missing param for `sortedValues`. 



##########
processing/src/main/java/org/apache/druid/query/filter/DimFilterUtils.java:
##########
@@ -62,6 +62,7 @@ public class DimFilterUtils
   static final byte RANGE_CACHE_ID = 0x14;
   static final byte IS_FILTER_BOOLEAN_FILTER_CACHE_ID = 0x15;
   static final byte ARRAY_CONTAINS_CACHE_ID = 0x16;
+  static final byte NEW_IN_CACHE_ID = 0x17;

Review Comment:
   ```suggestion
     static final byte TYPED_IN_CACHE_ID = 0x17;
   ```



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {

Review Comment:
   what do you think of validating that sortedValues are indeed sorted? 



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {

Review Comment:
   should you also check that both of them are not null either



##########
processing/src/main/java/org/apache/druid/segment/index/semantic/ValueSetIndexes.java:
##########
@@ -0,0 +1,273 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.index.semantic;
+
+import com.google.common.base.Supplier;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.PeekingIterator;
+import org.apache.druid.collections.bitmap.BitmapFactory;
+import org.apache.druid.collections.bitmap.ImmutableBitmap;
+import org.apache.druid.segment.column.TypeSignature;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.data.Indexed;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.SimpleImmutableBitmapDelegatingIterableIndex;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+
+public interface ValueSetIndexes

Review Comment:
   javadocs



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      this.unsortedValues = values;
+      this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());
+        for (Object s : lazyMatchValues.get()) {
+          matchValueBytes.add(StringUtils.toUtf8ByteBuffer(Evals.asString(s)));
+        }
+        return matchValueBytes;
+      });
+    } else {
+      this.lazyMatchValueBytes = null;
+    }
+
+    this.predicateFactorySupplier = Suppliers.memoize(
+        () -> new InFilterDruidPredicateFactory(lazyMatchValues.get(), matchValueType)
+    );
+    this.cacheKeySupplier = Suppliers.memoize(this::computeCacheKey);
+  }
+
+  @JsonProperty
+  public String getColumn()
+  {
+    return column;
+  }
+
+  @JsonProperty
+  public List<?> getSortedValues()
+  {
+    return lazyMatchValues.get();
+  }
+
+  @JsonProperty
+  public ColumnType getMatchValueType()
+  {
+    return matchValueType;
+  }
+
+  @Nullable
+  @JsonInclude(JsonInclude.Include.NON_NULL)
+  @JsonProperty
+  public FilterTuning getFilterTuning()
+  {
+    return filterTuning;
+  }
+
+  @Override
+  public byte[] getCacheKey()
+  {
+    return cacheKeySupplier.get();
+  }
+
+  @Override
+  public DimFilter optimize(final boolean mayIncludeUnknown)
+  {
+    final List<?> matchValues = lazyMatchValues.get();
+    if (matchValues.isEmpty()) {
+      return FalseDimFilter.instance();
+    } else if (matchValues.size() == 1) {
+      if (matchValues.get(0) == null) {
+        return NullFilter.forColumn(column);
+      }
+      return new EqualityFilter(
+          column,
+          matchValueType,
+          matchValues.iterator().next(),
+          filterTuning
+      );
+    }
+    return this;
+  }
+
+  @Override
+  public Filter toFilter()
+  {
+    return this;
+  }
+
+  @Nullable
+  @Override
+  public RangeSet<String> getDimensionRangeSet(String dimension)
+  {
+    if (!Objects.equals(getColumn(), dimension)) {
+      return null;
+    }
+    RangeSet<String> retSet = TreeRangeSet.create();
+    for (Object value : lazyMatchValues.get()) {
+      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));

Review Comment:
   since this filter only works in sql compatible mode, why wrap inside `NullHandling.nullToEmptyIfNeeded`? 



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      this.unsortedValues = values;
+      this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());
+        for (Object s : lazyMatchValues.get()) {
+          matchValueBytes.add(StringUtils.toUtf8ByteBuffer(Evals.asString(s)));
+        }
+        return matchValueBytes;
+      });
+    } else {
+      this.lazyMatchValueBytes = null;
+    }
+
+    this.predicateFactorySupplier = Suppliers.memoize(
+        () -> new InFilterDruidPredicateFactory(lazyMatchValues.get(), matchValueType)
+    );
+    this.cacheKeySupplier = Suppliers.memoize(this::computeCacheKey);
+  }
+
+  @JsonProperty
+  public String getColumn()
+  {
+    return column;
+  }
+
+  @JsonProperty
+  public List<?> getSortedValues()
+  {
+    return lazyMatchValues.get();
+  }
+
+  @JsonProperty
+  public ColumnType getMatchValueType()
+  {
+    return matchValueType;
+  }
+
+  @Nullable
+  @JsonInclude(JsonInclude.Include.NON_NULL)
+  @JsonProperty
+  public FilterTuning getFilterTuning()
+  {
+    return filterTuning;
+  }
+
+  @Override
+  public byte[] getCacheKey()
+  {
+    return cacheKeySupplier.get();
+  }
+
+  @Override
+  public DimFilter optimize(final boolean mayIncludeUnknown)
+  {
+    final List<?> matchValues = lazyMatchValues.get();
+    if (matchValues.isEmpty()) {
+      return FalseDimFilter.instance();
+    } else if (matchValues.size() == 1) {
+      if (matchValues.get(0) == null) {
+        return NullFilter.forColumn(column);
+      }
+      return new EqualityFilter(
+          column,
+          matchValueType,
+          matchValues.iterator().next(),
+          filterTuning
+      );
+    }
+    return this;
+  }
+
+  @Override
+  public Filter toFilter()
+  {
+    return this;
+  }
+
+  @Nullable
+  @Override
+  public RangeSet<String> getDimensionRangeSet(String dimension)
+  {
+    if (!Objects.equals(getColumn(), dimension)) {
+      return null;
+    }
+    RangeSet<String> retSet = TreeRangeSet.create();
+    for (Object value : lazyMatchValues.get()) {
+      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));
+      if (valueEquivalent == null) {
+        // Case when SQL compatible null handling is enabled
+        // Range.singleton(null) is invalid, so use the fact that
+        // only null values are less than empty string.
+        retSet.add(Range.lessThan(""));
+      } else {
+        retSet.add(Range.singleton(valueEquivalent));
+      }
+    }
+    return retSet;
+  }
+
+  @Override
+  public Set<String> getRequiredColumns()
+  {
+    return ImmutableSet.of(column);
+  }
+
+  @Override
+  @Nullable
+  public BitmapColumnIndex getBitmapColumnIndex(ColumnIndexSelector selector)
+  {
+    if (!Filters.checkFilterTuningUseIndex(column, selector, filterTuning)) {
+      return null;
+    }
+    final ColumnIndexSupplier indexSupplier = selector.getIndexSupplier(column);
+
+    if (indexSupplier == null) {
+      // column doesn't exist, match against null
+      DruidPredicateMatch match = predicateFactorySupplier.get().makeStringPredicate().apply(null);
+      return Filters.makeMissingColumnNullIndex(match, selector);
+    }
+
+    if (lazyMatchValueBytes != null) {
+      final Utf8ValueSetIndexes utf8ValueSetIndexes = indexSupplier.as(Utf8ValueSetIndexes.class);
+      if (utf8ValueSetIndexes != null) {
+        return utf8ValueSetIndexes.forSortedValuesUtf8(lazyMatchValueBytes.get());
+      }
+    }
+
+    final ValueSetIndexes valueSetIndexes = indexSupplier.as(ValueSetIndexes.class);
+    if (valueSetIndexes != null) {
+      return valueSetIndexes.forSortedValues(lazyMatchValues.get(), matchValueType);
+    }
+
+    return Filters.makePredicateIndex(
+        column,
+        selector,
+        predicateFactorySupplier.get()
+    );
+  }
+
+  @Override
+  public ValueMatcher makeMatcher(ColumnSelectorFactory factory)
+  {
+    return Filters.makeValueMatcher(factory, column, predicateFactorySupplier.get());
+  }
+
+  @Override
+  public VectorValueMatcher makeVectorMatcher(final VectorColumnSelectorFactory factory)
+  {
+    return ColumnProcessors.makeVectorProcessor(
+        column,
+        VectorValueMatcherColumnProcessorFactory.instance(),
+        factory
+    ).makeMatcher(predicateFactorySupplier.get());
+  }
+
+  @Override
+  public boolean canVectorizeMatcher(ColumnInspector inspector)
+  {
+    return true;
+  }
+
+  @Override
+  public boolean supportsRequiredColumnRewrite()
+  {
+    return true;
+  }
+
+  @Override
+  public Filter rewriteRequiredColumns(Map<String, String> columnRewrites)
+  {
+    String rewriteDimensionTo = columnRewrites.get(column);
+    if (rewriteDimensionTo == null) {
+      throw new IAE("Received a non-applicable rewrite: %s, filter's dimension: %s", columnRewrites, column);
+    }
+
+    if (rewriteDimensionTo.equals(column)) {
+      return this;
+    } else {
+      return new TypedInFilter(
+          rewriteDimensionTo,
+          matchValueType,
+          null,
+          lazyMatchValues.get(),
+          filterTuning
+      );
+    }
+  }
+
+  @Override
+  public String toString()
+  {
+    final DimFilter.DimFilterToStringBuilder builder = new DimFilter.DimFilterToStringBuilder();
+    return builder.appendDimension(column, null)
+                  .append(" IN (")
+                  .append(Joiner.on(", ").join(Iterables.transform(lazyMatchValues.get(), String::valueOf)))
+                  .append(")")
+                  .append(" (" + matchValueType + ")")
+                  .appendFilterTuning(filterTuning)
+                  .build();
+  }
+
+  @Override
+  public boolean equals(Object o)
+  {
+    if (this == o) {
+      return true;
+    }
+    if (o == null || getClass() != o.getClass()) {
+      return false;
+    }
+    TypedInFilter that = (TypedInFilter) o;
+    return column.equals(that.column) &&
+           Objects.equals(matchValueType, that.matchValueType) &&
+           compareValues(lazyMatchValues.get(), that.lazyMatchValues.get(), matchValueType) &&
+           Objects.equals(filterTuning, that.filterTuning);
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Objects.hash(lazyMatchValues.get(), column, matchValueType, filterTuning);
+  }
+
+  private byte[] computeCacheKey()
+  {
+    // Hash all values, in sorted order, as their length followed by their content.
+    final Hasher hasher = Hashing.sha256().newHasher();
+    for (Object v : lazyMatchValues.get()) {
+      if (v == null) {
+        // Encode null as length -1, no content.
+        hasher.putInt(-1);
+      } else {
+        final String s = Evals.asString(v);
+        hasher.putInt(s.length());
+        hasher.putString(s, StandardCharsets.UTF_8);
+      }
+    }
+
+    return new CacheKeyBuilder(DimFilterUtils.NEW_IN_CACHE_ID)
+        .appendString(column)
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendString(matchValueType.asTypeString())
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendByteArray(hasher.hash().asBytes())
+        .build();
+  }
+
+  private static List<?> sortValues(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Stream<?> coerced;
+    if (matchValueType.is(ValueType.LONG)) {
+      coerced = unsortedValues.stream().map(DimensionHandlerUtils::convertObjectToLong).distinct();
+    } else if (matchValueType.is(ValueType.DOUBLE)) {
+      coerced = unsortedValues.stream().map(DimensionHandlerUtils::convertObjectToDouble).distinct();
+    } else if (matchValueType.is(ValueType.FLOAT)) {
+      coerced = unsortedValues.stream().map(DimensionHandlerUtils::convertObjectToFloat).distinct();
+    } else {
+      coerced = unsortedValues.stream().distinct();
+    }
+    return coerced.sorted(matchValueType.getNullableStrategy()).collect(Collectors.toList());
+  }
+
+  /**
+   * Since jackson might translate longs into ints and such, we use the type comparator to check lists
+   * for {@link #equals(Object)} instead of directly using {@link Objects#equals(Object, Object)}
+   */
+  private static boolean compareValues(List<?> o1, List<?> o2, ColumnType matchValueType)
+  {
+    final NullableTypeStrategy comparator = matchValueType.getNullableStrategy();
+    //noinspection ObjectEquality
+    if (o1 == o2) {
+      return true;
+    }
+    if (o1 == null) {
+      return false;
+    }
+    if (o2 == null) {
+      return false;
+    }
+    final int iter = Math.min(o1.size(), o2.size());
+    for (int i = 0; i < iter; i++) {
+      final int cmp = comparator.compare(o1.get(i), o2.get(i));
+      if (cmp == 0) {
+        continue;
+      }
+      return false;
+    }
+    return o1.size() == o2.size();

Review Comment:
   could be done before the loop to make the comparison faster. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533188709


##########
processing/src/main/java/org/apache/druid/segment/nested/ScalarDoubleColumnAndIndexSupplier.java:
##########
@@ -258,6 +265,70 @@ public <T> T computeBitmapResult(BitmapResultFactory<T> bitmapResultFactory, boo
     }
   }
 
+  private final class DoubleValueSetIndexes implements ValueSetIndexes
+  {
+    final FixedIndexed<Double> dictionary = doubleDictionarySupplier.get();
+
+    @Nullable
+    @Override
+    public BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType)
+    {
+      final boolean matchNull = sortedValues.get(0) == null;
+      final Supplier<ImmutableBitmap> unknownsIndex = () -> {
+        if (!matchNull && dictionary.get(0) == null) {
+          return valueIndexes.get(0);
+        }
+        return null;
+      };
+
+      // values are doubles and ordered in double order
+      if (matchValueType.is(ValueType.DOUBLE)) {

Review Comment:
   yea probably, also for the other index implementations.. i can look into it since has been on my todo list for a while



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533194261


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {

Review Comment:
   oops, `checkSorted` actually does mark it as not sorted if there are dupe values, but `sortValues` method used to dedupe and now it isn't, i should probably fix that, was lost when i switched to using `ObjectArrays.quickSort`.
   
   that said, dupes wouldn't really break anything, just inefficient



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1533196664


##########
extensions-core/druid-ranger-security/pom.xml:
##########
@@ -45,6 +45,7 @@
                 <groupId>com.amazonaws</groupId>
                 <artifactId>aws-java-sdk-bundle</artifactId>
                 <version>${aws.sdk.version}</version>
+                <scope>provided</scope>

Review Comment:
   comment is outdated, iirc this was an accident from experimenting with something else in the wrong branch



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534871515


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {
+        this.unsortedValues = null;
+        this.lazyMatchValues = () -> values;
+      } else {
+        this.unsortedValues = values;
+        this.lazyMatchValues = Suppliers.memoize(() -> sortValues(unsortedValues, matchValueType));
+      }
+    }
+    if (matchValueType.is(ValueType.STRING)) {
+      this.lazyMatchValueBytes = Suppliers.memoize(() -> {
+        final SortedSet<ByteBuffer> matchValueBytes = new ObjectAVLTreeSet<>(ByteBufferUtils.utf8Comparator());
+        for (Object s : lazyMatchValues.get()) {
+          matchValueBytes.add(StringUtils.toUtf8ByteBuffer(Evals.asString(s)));
+        }
+        return matchValueBytes;
+      });
+    } else {
+      this.lazyMatchValueBytes = null;
+    }
+
+    this.predicateFactorySupplier = Suppliers.memoize(
+        () -> new PredicateFactory(lazyMatchValues.get(), matchValueType)
+    );
+    this.cacheKeySupplier = Suppliers.memoize(this::computeCacheKey);
+  }
+
+  @JsonProperty
+  public String getColumn()
+  {
+    return column;
+  }
+
+  @JsonProperty
+  public List<?> getSortedValues()
+  {
+    return lazyMatchValues.get();
+  }
+
+  @JsonProperty
+  public ColumnType getMatchValueType()
+  {
+    return matchValueType;
+  }
+
+  @Nullable
+  @JsonInclude(JsonInclude.Include.NON_NULL)
+  @JsonProperty
+  public FilterTuning getFilterTuning()
+  {
+    return filterTuning;
+  }
+
+  @Override
+  public byte[] getCacheKey()
+  {
+    return cacheKeySupplier.get();
+  }
+
+  @Override
+  public DimFilter optimize(final boolean mayIncludeUnknown)
+  {
+    final List<?> matchValues = lazyMatchValues.get();
+    if (matchValues.isEmpty()) {
+      return FalseDimFilter.instance();
+    } else if (matchValues.size() == 1) {
+      if (matchValues.get(0) == null) {
+        return NullFilter.forColumn(column);
+      }
+      return new EqualityFilter(
+          column,
+          matchValueType,
+          matchValues.iterator().next(),
+          filterTuning
+      );
+    }
+    return this;
+  }
+
+  @Override
+  public Filter toFilter()
+  {
+    return this;
+  }
+
+  @Nullable
+  @Override
+  public RangeSet<String> getDimensionRangeSet(String dimension)
+  {
+    if (!Objects.equals(getColumn(), dimension)) {
+      return null;
+    }
+    RangeSet<String> retSet = TreeRangeSet.create();
+    for (Object value : lazyMatchValues.get()) {
+      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));
+      if (valueEquivalent == null) {
+        // Case when SQL compatible null handling is enabled
+        // Range.singleton(null) is invalid, so use the fact that
+        // only null values are less than empty string.
+        retSet.add(Range.lessThan(""));
+      } else {
+        retSet.add(Range.singleton(valueEquivalent));
+      }
+    }
+    return retSet;
+  }
+
+  @Override
+  public Set<String> getRequiredColumns()
+  {
+    return ImmutableSet.of(column);
+  }
+
+  @Override
+  @Nullable
+  public BitmapColumnIndex getBitmapColumnIndex(ColumnIndexSelector selector)
+  {
+    if (!Filters.checkFilterTuningUseIndex(column, selector, filterTuning)) {
+      return null;
+    }
+    final ColumnIndexSupplier indexSupplier = selector.getIndexSupplier(column);
+
+    if (indexSupplier == null) {
+      // column doesn't exist, match against null
+      DruidPredicateMatch match = predicateFactorySupplier.get().makeStringPredicate().apply(null);
+      return Filters.makeMissingColumnNullIndex(match, selector);
+    }
+
+    if (lazyMatchValueBytes != null) {
+      final Utf8ValueSetIndexes utf8ValueSetIndexes = indexSupplier.as(Utf8ValueSetIndexes.class);
+      if (utf8ValueSetIndexes != null) {
+        return utf8ValueSetIndexes.forSortedValuesUtf8(lazyMatchValueBytes.get());
+      }
+    }
+
+    final ValueSetIndexes valueSetIndexes = indexSupplier.as(ValueSetIndexes.class);
+    if (valueSetIndexes != null) {
+      return valueSetIndexes.forSortedValues(lazyMatchValues.get(), matchValueType);
+    }
+
+    return Filters.makePredicateIndex(
+        column,
+        selector,
+        predicateFactorySupplier.get()
+    );
+  }
+
+  @Override
+  public ValueMatcher makeMatcher(ColumnSelectorFactory factory)
+  {
+    return Filters.makeValueMatcher(factory, column, predicateFactorySupplier.get());
+  }
+
+  @Override
+  public VectorValueMatcher makeVectorMatcher(final VectorColumnSelectorFactory factory)
+  {
+    return ColumnProcessors.makeVectorProcessor(
+        column,
+        VectorValueMatcherColumnProcessorFactory.instance(),
+        factory
+    ).makeMatcher(predicateFactorySupplier.get());
+  }
+
+  @Override
+  public boolean canVectorizeMatcher(ColumnInspector inspector)
+  {
+    return true;
+  }
+
+  @Override
+  public boolean supportsRequiredColumnRewrite()
+  {
+    return true;
+  }
+
+  @Override
+  public Filter rewriteRequiredColumns(Map<String, String> columnRewrites)
+  {
+    String rewriteDimensionTo = columnRewrites.get(column);
+    if (rewriteDimensionTo == null) {
+      throw new IAE("Received a non-applicable rewrite: %s, filter's dimension: %s", columnRewrites, column);
+    }
+
+    if (rewriteDimensionTo.equals(column)) {
+      return this;
+    } else {
+      return new TypedInFilter(
+          rewriteDimensionTo,
+          matchValueType,
+          null,
+          lazyMatchValues.get(),
+          filterTuning
+      );
+    }
+  }
+
+  @Override
+  public String toString()
+  {
+    final DimFilter.DimFilterToStringBuilder builder = new DimFilter.DimFilterToStringBuilder();
+    return builder.appendDimension(column, null)
+                  .append(" IN (")
+                  .append(Joiner.on(", ").join(Iterables.transform(lazyMatchValues.get(), String::valueOf)))
+                  .append(")")
+                  .append(" (" + matchValueType + ")")
+                  .appendFilterTuning(filterTuning)
+                  .build();
+  }
+
+  @Override
+  public boolean equals(Object o)
+  {
+    if (this == o) {
+      return true;
+    }
+    if (o == null || getClass() != o.getClass()) {
+      return false;
+    }
+    TypedInFilter that = (TypedInFilter) o;
+    return column.equals(that.column) &&
+           Objects.equals(matchValueType, that.matchValueType) &&
+           compareValues(lazyMatchValues.get(), that.lazyMatchValues.get(), matchValueType) &&
+           Objects.equals(filterTuning, that.filterTuning);
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Objects.hash(lazyMatchValues.get(), column, matchValueType, filterTuning);
+  }
+
+  private byte[] computeCacheKey()
+  {
+    // Hash all values, in sorted order, as their length followed by their content.
+    final Hasher hasher = Hashing.sha256().newHasher();
+    for (Object v : lazyMatchValues.get()) {
+      if (v == null) {
+        // Encode null as length -1, no content.
+        hasher.putInt(-1);
+      } else {
+        final String s = Evals.asString(v);
+        hasher.putInt(s.length());
+        hasher.putString(s, StandardCharsets.UTF_8);
+      }
+    }
+
+    return new CacheKeyBuilder(DimFilterUtils.NEW_IN_CACHE_ID)
+        .appendString(column)
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendString(matchValueType.asTypeString())
+        .appendByte(DimFilterUtils.STRING_SEPARATOR)
+        .appendByteArray(hasher.hash().asBytes())
+        .build();
+  }
+
+  private static boolean checkSorted(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Comparator<Object> comparator = matchValueType.getNullableStrategy();
+    Object prev = null;
+    boolean needsCoerceCheck = true;
+    for (Object o : unsortedValues) {
+      if (needsCoerceCheck && o != null) {
+        Object coerced = coerceValue(o, matchValueType);
+        //noinspection ObjectEquality
+        if (coerced != o) {
+          return false;
+        }
+        needsCoerceCheck = false;
+      }
+      if (prev != null && comparator.compare(prev, o) >= 0) {
+        return false;
+      }
+      prev = o;
+    }
+    return true;
+  }
+
+  @Nullable
+  private static Object coerceValue(@Nullable Object o, ColumnType matchValueType)
+  {
+    if (o == null) {
+      return o;
+    }
+    switch (matchValueType.getType()) {
+      case STRING:
+        return DimensionHandlerUtils.convertObjectToString(o);
+      case LONG:
+        return DimensionHandlerUtils.convertObjectToLong(o);
+      case FLOAT:
+        return DimensionHandlerUtils.convertObjectToFloat(o);
+      case DOUBLE:
+        return DimensionHandlerUtils.convertObjectToDouble(o);
+      default:
+        throw InvalidInput.exception("Unsupported matchValueType[%s]", matchValueType);
+    }
+  }
+
+  private static List<?> sortValues(List<?> unsortedValues, ColumnType matchValueType)
+  {
+    final Object[] array = unsortedValues.toArray(new Object[0]);
+    // check if values need coerced
+    for (int i = 0; i < array.length; i++) {
+      Object coerced = coerceValue(array[i], matchValueType);
+      //noinspection ObjectEquality
+      if (coerced != null && array[i] == coerced) {
+        // assume list is all same type objects...

Review Comment:
   added coercion while sorting just in case



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1534865608


##########
processing/src/main/java/org/apache/druid/segment/nested/ScalarDoubleColumnAndIndexSupplier.java:
##########
@@ -258,6 +265,70 @@ public <T> T computeBitmapResult(BitmapResultFactory<T> bitmapResultFactory, boo
     }
   }
 
+  private final class DoubleValueSetIndexes implements ValueSetIndexes
+  {
+    final FixedIndexed<Double> dictionary = doubleDictionarySupplier.get();
+
+    @Nullable
+    @Override
+    public BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType)
+    {
+      final boolean matchNull = sortedValues.get(0) == null;
+      final Supplier<ImmutableBitmap> unknownsIndex = () -> {
+        if (!matchNull && dictionary.get(0) == null) {
+          return valueIndexes.get(0);
+        }
+        return null;
+      };
+
+      // values are doubles and ordered in double order
+      if (matchValueType.is(ValueType.DOUBLE)) {

Review Comment:
   i have not done this change yet, still thinking about it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1535024793


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -222,12 +254,9 @@ public RangeSet<String> getDimensionRangeSet(String dimension)
       return null;
     }
     RangeSet<String> retSet = TreeRangeSet.create();
-    for (Object value : lazyMatchValues.get()) {
-      String valueEquivalent = NullHandling.nullToEmptyIfNeeded(Evals.asString(value));
+    for (Object value : sortedMatchValues.get()) {
+      String valueEquivalent = Evals.asString(value);
       if (valueEquivalent == null) {
-        // Case when SQL compatible null handling is enabled
-        // Range.singleton(null) is invalid, so use the fact that

Review Comment:
   Most of this comment is still useful in explaining why we're doing `lessThan("")`.



##########
processing/src/main/java/org/apache/druid/segment/nested/ScalarDoubleColumnAndIndexSupplier.java:
##########
@@ -258,6 +265,70 @@ public <T> T computeBitmapResult(BitmapResultFactory<T> bitmapResultFactory, boo
     }
   }
 
+  private final class DoubleValueSetIndexes implements ValueSetIndexes
+  {
+    final FixedIndexed<Double> dictionary = doubleDictionarySupplier.get();
+
+    @Nullable
+    @Override
+    public BitmapColumnIndex forSortedValues(@Nonnull List<?> sortedValues, TypeSignature<ValueType> matchValueType)
+    {
+      final boolean matchNull = sortedValues.get(0) == null;
+      final Supplier<ImmutableBitmap> unknownsIndex = () -> {
+        if (!matchNull && dictionary.get(0) == null) {
+          return valueIndexes.get(0);
+        }
+        return null;
+      };
+
+      // values are doubles and ordered in double order
+      if (matchValueType.is(ValueType.DOUBLE)) {

Review Comment:
   I am okay with what you decide here. I can see both ways making sense.



##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,742 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectRBTreeSet;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * Approximately like the SQL 'IN' filter, with the main difference being that this will match NULL values if contained
+ * in the values list instead of ignoring them.
+ * <p>
+ * This is a typed version of {@link InDimFilter}, which allows the match values to exist in their native type and
+ * sorted in their type native order for better performance matching against all column types.
+ */
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  /**
+   * Column to match {@link #sortedMatchValues} or {@link #sortedUtf8MatchValueBytes} against.
+   */
+  private final String column;
+
+  /**
+   * Type of values contained in {@link #sortedMatchValues}. This might be the same or different than the
+   * {@link ColumnType} of {@link #column}, but is encouraged to be the same there are several optimizations available
+   * if they match.
+   */
+  private final ColumnType matchValueType;
+
+  /**
+   * Unsorted values. This will be null if the values are found to be sorted, or have been already sorted "upstream".
+   * Otherwise, this set of values will be lazily computed into {@link #sortedMatchValues} as needed, e.g. for
+   * JSON serialization, cache key building, building a hashcode, or checking equality.
+   */
+  @Nullable
+  private final List<?> unsortedValues;
+
+  /**
+   * Supplier for list of values sorted by {@link #matchValueType}. This is lazily computed if
+   * {@link #unsortedValues} is not null and previously sorted.

Review Comment:
   would be useful to include a comment about whether this can contain duplicates or not, and if duplicates are not meant to be here, then also include a test that validates that deduplication happens on `values` (even if they are provided in sorted order).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] add new typed in filter (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16039:
URL: https://github.com/apache/druid/pull/16039#discussion_r1535030629


##########
processing/src/main/java/org/apache/druid/query/filter/TypedInFilter.java:
##########
@@ -0,0 +1,713 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.filter;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Range;
+import com.google.common.collect.RangeSet;
+import com.google.common.collect.Sets;
+import com.google.common.collect.TreeRangeSet;
+import com.google.common.hash.Hasher;
+import com.google.common.hash.Hashing;
+import it.unimi.dsi.fastutil.doubles.DoubleOpenHashSet;
+import it.unimi.dsi.fastutil.floats.FloatOpenHashSet;
+import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
+import it.unimi.dsi.fastutil.objects.ObjectAVLTreeSet;
+import it.unimi.dsi.fastutil.objects.ObjectArrays;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.error.InvalidInput;
+import org.apache.druid.java.util.common.ByteBufferUtils;
+import org.apache.druid.java.util.common.IAE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.math.expr.Evals;
+import org.apache.druid.query.cache.CacheKeyBuilder;
+import org.apache.druid.query.filter.vector.VectorValueMatcher;
+import org.apache.druid.query.filter.vector.VectorValueMatcherColumnProcessorFactory;
+import org.apache.druid.segment.ColumnInspector;
+import org.apache.druid.segment.ColumnProcessors;
+import org.apache.druid.segment.ColumnSelectorFactory;
+import org.apache.druid.segment.DimensionHandlerUtils;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.column.ColumnType;
+import org.apache.druid.segment.column.NullableTypeStrategy;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.filter.Filters;
+import org.apache.druid.segment.index.BitmapColumnIndex;
+import org.apache.druid.segment.index.semantic.Utf8ValueSetIndexes;
+import org.apache.druid.segment.index.semantic.ValueSetIndexes;
+import org.apache.druid.segment.vector.VectorColumnSelectorFactory;
+
+import javax.annotation.Nullable;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.SortedSet;
+
+public class TypedInFilter extends AbstractOptimizableDimFilter implements Filter
+{
+  private final String column;
+  private final ColumnType matchValueType;
+  @Nullable
+  private final List<?> unsortedValues;
+  private final Supplier<List<?>> lazyMatchValues;
+  @Nullable
+  private final Supplier<SortedSet<ByteBuffer>> lazyMatchValueBytes;
+  @Nullable
+  private final FilterTuning filterTuning;
+  private final Supplier<DruidPredicateFactory> predicateFactorySupplier;
+
+  @JsonIgnore
+  private final Supplier<byte[]> cacheKeySupplier;
+
+  /**
+   * Creates a new filter.
+   *
+   * @param column         column to search
+   * @param values         set of values to match. This collection may be reused to avoid copying a big collection.
+   *                       Therefore, callers should <b>not</b> modify the collection after it is passed to this
+   *                       constructor.
+   * @param matchValueType type of values contained in set
+   * @param filterTuning   optional tuning
+   */
+  @JsonCreator
+  public TypedInFilter(
+      @JsonProperty("column") String column,
+      @JsonProperty("matchValueType") ColumnType matchValueType,
+      @JsonProperty("values") @Nullable List<?> values,
+      @JsonProperty("sortedValues") @Nullable List<?> sortedValues,
+      @JsonProperty("filterTuning") @Nullable FilterTuning filterTuning
+  )
+  {
+    if (NullHandling.replaceWithDefault()) {
+      throw InvalidInput.exception(
+          "Invalid IN filter, typed in filter only supports SQL compatible null handling mode, set druid.generic.useDefaultValue=false to use this filter"
+      );
+    }
+    this.column = column;
+    if (column == null) {
+      throw InvalidInput.exception("Invalid IN filter, column cannot be null");
+    }
+    this.filterTuning = filterTuning;
+    this.matchValueType = matchValueType;
+    if (matchValueType == null) {
+      throw InvalidInput.exception("Invalid IN filter on column [%s], matchValueType cannot be null", column);
+    }
+    // one of sorted or not sorted
+    if (values == null && sortedValues == null) {
+      throw InvalidInput.exception(
+          "Invalid IN filter on column [%s], exactly one of values or sortedValues must be non-null",
+          column
+      );
+    }
+    if (sortedValues != null) {
+      this.unsortedValues = null;
+      this.lazyMatchValues = () -> sortedValues;
+    } else {
+      if (checkSorted(values, matchValueType)) {

Review Comment:
   yea, i toyed with a bit but didn't spend anytime measuring anything yet. I think the main thing that made me switch was the addition of the value coercion, with it too we iterate list once to make them all be the matchValueType, then the sort, then again to dedupe, so it seemed like maybe i should just use the set and coerce on insert and then turn that into a list.
   
   Now i'm a bit curious though, so will do some measurements.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org