You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/08/17 11:49:32 UTC

[GitHub] [druid] clintropolis opened a new pull request, #12914: tighten up array handling, fix bug with array_slice output type inference

clintropolis opened a new pull request, #12914:
URL: https://github.com/apache/druid/pull/12914

   ### Description
   This PR adjusts array handling in the direction of standardizing the representations on `Object[]` as well as `long[]`, `double[]`, and `float[]`. This is follow-up to https://github.com/apache/druid/pull/12498#discussion_r941829087 and a follow-up discussion I had with @gianm.
   
   The main changes are removing the remaining leaks of typed object arrays such as `Long[]`, `Double[]`, and `Float[]` from the expression system, as well as adding support for handling arrays of java primitives such as `long[]`, `float[]`, and `double[]`. Expressions do not produce these primitive arrays, but aggregators and post-aggs can.
   
   I've added javadocs to try to help provide additional details on native type matters.
   
   I've adjusted the behavior of the expression column value selector when the virtual column output type is `STRING` to behave a bit more like a normal multi-value string column, and return the value directly when there is only a single value contained in the list. This does slightly change scan query results, which for multi-value string expressions would always produce array typed results, but now will produce a mix of array and scalar values. I think this behavior is more consistent with how the multi-value string columns themselves behave, but I can put it behind a flag if anyone is worried about this change.
   
   I encountered a bug with `array_slice` output type inference, which was not indicating its output as array output, instead reporting the output type of its argument. This issue became clear due to the stronger separation between multi-value string handling and array handling that now occurs in the expression selectors.
   
   <hr>
   
   
   This PR has:
   - [ ] been self-reviewed.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953748084


##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a
+   * {@link java.util.List<String>}.
    */
   STRING,
   /**
    * Placeholder for arbitrary 'complex' types, which have a corresponding serializer/deserializer implementation. Note
    * that knowing a type is complex alone isn't enough information to work with it directly, and additional information
-   * in the form of a type name that is registered in the complex type registry must be available to make this type
-   * meaningful. This type is not currently supported as a grouping key for aggregations, and may not be used as an
-   * input to expression virtual columns, and might only be supported by the specific aggregators crafted to handle
-   * this complex type.
+   * in the form of a type name which must be registered in the complex type registry. This type is not currently
+   * supported as a grouping key for aggregations, and might only be supported by the specific aggregators crafted to
+   * handle this complex type. Filtering on this type with standard filters will most likely have limited support, and

Review Comment:
   this seemed too complicated to explain so now just mention that filters will treat as null



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953256771


##########
core/src/main/java/org/apache/druid/segment/column/ColumnType.java:
##########
@@ -26,18 +26,71 @@
 
 import javax.annotation.Nullable;
 
+/**
+ * Native Druid types.
+ *
+ * @see TypeSignature
+ */
 @JsonSerialize(using = ToStringSerializer.class)
 public class ColumnType extends BaseTypeSignature<ValueType>
 {
+  /**
+   * Druid string type. Values will be represented as {@link String} or {@link java.util.List<String>} in the case
+   * of multi-value string columns. {@link ColumnType} has insufficient information to distinguish between single
+   * and multi-value strings, this requires a type 'ColumnCapabilities' which is available at a higher layer.

Review Comment:
   not in this package unfortunately but will do. i do still someday wish to merge druid-core into druid-processing since i find the split a bit frustrating, but that is so disruptive i should probably give up hope that it will ever happen :trollface: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953255543


##########
core/src/main/java/org/apache/druid/math/expr/Evals.java:
##########
@@ -20,21 +20,22 @@
 package org.apache.druid.math.expr;
 
 import org.apache.druid.common.config.NullHandling;
-import org.apache.druid.java.util.common.logger.Logger;
 
 import javax.annotation.Nullable;
-import java.util.Arrays;
 import java.util.List;
 
 /**
  */
 public class Evals
 {
-  private static final Logger log = new Logger(Evals.class);
-
   public static boolean isAllConstants(Expr... exprs)
   {
-    return isAllConstants(Arrays.asList(exprs));
+    for (Expr expr : exprs) {

Review Comment:
   yeah, this change wasn't really related to anything, but it seemed a bit pointless to wrap it... though i guess probably not that expensive in the scheme of things since this is only used when flattening expressions in the `Parser`, so on second thought can just change it back



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953257381


##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a

Review Comment:
   I guess i intended to convey that it can vary from value to value/row to row, but on second reading agree its not entirely clear, will make some adjustments



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953261475


##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a
+   * {@link java.util.List<String>}.
    */
   STRING,
   /**
    * Placeholder for arbitrary 'complex' types, which have a corresponding serializer/deserializer implementation. Note
    * that knowing a type is complex alone isn't enough information to work with it directly, and additional information
-   * in the form of a type name that is registered in the complex type registry must be available to make this type
-   * meaningful. This type is not currently supported as a grouping key for aggregations, and may not be used as an
-   * input to expression virtual columns, and might only be supported by the specific aggregators crafted to handle
-   * this complex type.
+   * in the form of a type name which must be registered in the complex type registry. This type is not currently
+   * supported as a grouping key for aggregations, and might only be supported by the specific aggregators crafted to
+   * handle this complex type. Filtering on this type with standard filters will most likely have limited support, and

Review Comment:
   so, there is [`isFilterable`](https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/column/ColumnCapabilities.java#L95) on column capabilities which is used exclusively to allow complex types to provide indexes to use for filtering, https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/ColumnSelectorColumnIndexSelector.java#L86. This could allow complex columns to directly provide indexes that filters understand, which I suppose could be used for filtering.
   
   However, there are no examples of this in current complex types, and importantly, I don't consider this support to be fully built-out yet though because without value matcher integration its not really complete and likely not correct except in certain situations. We did start to orient some things to make this someday be possible, like adding the object predicate https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/query/filter/DruidPredicateFactory.java#L46, but some more plumbing needs to be done to make use of it I think.
   
   The big win here on finishing this stuff out someday would be to allow matching constant forms of complex values with selector filters, and making is null/is not null actually work correctly (like is null matches everything since the column is treated as a null) which is like a special case of the same thing but especially jarring.
   
   Beyond equality checks I'm not sure most of the standard filters make sense with complex types in general, but tangentially, it would probably be nice in the future to have a mode where filtering on them with filters that don't make sense is an error condition instead of silently treating it as a column of null values.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953946774


##########
core/src/main/java/org/apache/druid/segment/column/ColumnType.java:
##########
@@ -26,18 +26,71 @@
 
 import javax.annotation.Nullable;
 
+/**
+ * Native Druid types.
+ *
+ * @see TypeSignature
+ */
 @JsonSerialize(using = ToStringSerializer.class)
 public class ColumnType extends BaseTypeSignature<ValueType>
 {
+  /**
+   * Druid string type. Values will be represented as {@link String} or {@link java.util.List<String>} in the case
+   * of multi-value string columns. {@link ColumnType} has insufficient information to distinguish between single
+   * and multi-value strings, this requires a type 'ColumnCapabilities' which is available at a higher layer.

Review Comment:
   > so disruptive i should probably give up hope that it will ever happen
   
   There's always the old "do it during a quiet week" strategy 🙂



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] lgtm-com[bot] commented on pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on PR #12914:
URL: https://github.com/apache/druid/pull/12914#issuecomment-1217994690

   This pull request **introduces 4 alerts** when merging 8763c668ee0773a5b46aedf567b47e01b231915b into f70f7b4b89a4b2d49a4a994541a16ea31d19fbac - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-eeab855cae2646f74159462d750a5c62bf541e52)
   
   **new alerts:**
   
   * 4 for Dereferenced variable may be null


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953262388


##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a
+   * {@link java.util.List<String>}.
    */
   STRING,
   /**
    * Placeholder for arbitrary 'complex' types, which have a corresponding serializer/deserializer implementation. Note
    * that knowing a type is complex alone isn't enough information to work with it directly, and additional information
-   * in the form of a type name that is registered in the complex type registry must be available to make this type
-   * meaningful. This type is not currently supported as a grouping key for aggregations, and may not be used as an
-   * input to expression virtual columns, and might only be supported by the specific aggregators crafted to handle
-   * this complex type.
+   * in the form of a type name which must be registered in the complex type registry. This type is not currently
+   * supported as a grouping key for aggregations, and might only be supported by the specific aggregators crafted to
+   * handle this complex type. Filtering on this type with standard filters will most likely have limited support, and
+   * may match as a "null" value.
+   *
+   * These types are represented by the individual Java type associated with the complex type name as defined in the
+   * type registry.
    */
   COMPLEX,
 
   /**
-   * Placeholder for arbitrary arrays of other {@link ValueType}. This type is not currently supported as a grouping
+   * Placeholder for arbitrary arrays of other {@link ValueType}. This type has limited support as a grouping

Review Comment:
   you can group on specific array types, such as `ARRAY<STRING>`, `ARRAY<LONG>`, `ARRAY<DOUBLE>`, `ARRAY<FLOAT>`, but not fully generically, like on `ARRAY<ARRAY<LONG>>`, or `ARRAY<COMPLEX<json>>` (assuming grouping on `COMPLEX` types was also supported, which it isn't yet... but soon.). I'll clarify that some specific array grouping is supported, but array grouping in general is not.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
gianm commented on PR #12914:
URL: https://github.com/apache/druid/pull/12914#issuecomment-1224997884

   > 4 for Dereferenced variable may be null
   
   Do these matter?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis merged pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
clintropolis merged PR #12914:
URL: https://github.com/apache/druid/pull/12914


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953237605


##########
core/src/main/java/org/apache/druid/math/expr/ExprEval.java:
##########
@@ -429,18 +443,15 @@ public static ExprEval ofType(@Nullable ExpressionType type, @Nullable Object va
       case STRING:
         // not all who claim to be "STRING" are always a String, prepare ourselves...
         if (value instanceof String[]) {
-          return new ArrayExprEval(ExpressionType.STRING_ARRAY, (String[]) value);
+          return new ArrayExprEval(ExpressionType.STRING_ARRAY, Arrays.stream((String[]) value).toArray());

Review Comment:
   Stream construction in hot paths can be expensive. May be better to avoid it.



##########
core/src/main/java/org/apache/druid/math/expr/ExprEval.java:
##########
@@ -388,16 +388,30 @@ public static ExprEval bestEffortOf(@Nullable Object val)
       return new StringExprEval(String.valueOf(val));
     }
     if (val instanceof Long[]) {
-      return new ArrayExprEval(ExpressionType.LONG_ARRAY, (Long[]) val);
+      return new ArrayExprEval(ExpressionType.LONG_ARRAY, Arrays.stream((Long[]) val).toArray());
+    }
+    if (val instanceof long[]) {
+      return new ArrayExprEval(ExpressionType.LONG_ARRAY, Arrays.stream((long[]) val).boxed().toArray());
     }
     if (val instanceof Double[]) {
-      return new ArrayExprEval(ExpressionType.DOUBLE_ARRAY, (Double[]) val);
+      return new ArrayExprEval(ExpressionType.DOUBLE_ARRAY, Arrays.stream((Double[]) val).toArray());
+    }
+    if (val instanceof double[]) {
+      return new ArrayExprEval(ExpressionType.DOUBLE_ARRAY, Arrays.stream((double[]) val).boxed().toArray());
     }
     if (val instanceof Float[]) {
       return new ArrayExprEval(ExpressionType.DOUBLE_ARRAY, Arrays.stream((Float[]) val).map(Float::doubleValue).toArray());
     }
+    if (val instanceof float[]) {
+      float[] floats = (float[]) val;
+      Object[] boxed = new Object[floats.length];
+      for (int i = 0; i < floats.length; i++) {
+        boxed[i] = floats[i];
+      }
+      return new ArrayExprEval(ExpressionType.DOUBLE_ARRAY, boxed);
+    }
     if (val instanceof String[]) {
-      return new ArrayExprEval(ExpressionType.STRING_ARRAY, (String[]) val);
+      return new ArrayExprEval(ExpressionType.STRING_ARRAY, Arrays.stream((String[]) val).toArray());

Review Comment:
   
   
   Stream construction in hot paths can be expensive. May be better to avoid it.
   



##########
processing/src/main/java/org/apache/druid/segment/filter/ExpressionFilter.java:
##########
@@ -141,26 +141,26 @@ public boolean matches()
         if (eval.type().isArray()) {
           switch (eval.elementType().getType()) {
             case LONG:
-              final Long[] lResult = eval.asLongArray();
+              final Object[] lResult = eval.asArray();
               if (lResult == null) {
                 return false;
               }
 
-              return Arrays.stream(lResult).filter(Objects::nonNull).anyMatch(Evals::asBoolean);
+              return Arrays.stream(lResult).filter(Objects::nonNull).anyMatch(o -> Evals.asBoolean((long) o));

Review Comment:
   I know this code was pre-existing, but, may be a good idea to move it away from streams since it's a hot path.



##########
core/src/main/java/org/apache/druid/segment/column/ColumnType.java:
##########
@@ -26,18 +26,71 @@
 
 import javax.annotation.Nullable;
 
+/**
+ * Native Druid types.
+ *
+ * @see TypeSignature
+ */
 @JsonSerialize(using = ToStringSerializer.class)
 public class ColumnType extends BaseTypeSignature<ValueType>
 {
+  /**
+   * Druid string type. Values will be represented as {@link String} or {@link java.util.List<String>} in the case
+   * of multi-value string columns. {@link ColumnType} has insufficient information to distinguish between single
+   * and multi-value strings, this requires a type 'ColumnCapabilities' which is available at a higher layer.
+   *
+   * @see ValueType#STRING
+   */
   public static final ColumnType STRING = new ColumnType(ValueType.STRING, null, null);
+
+  /**
+   * Druid 64-bit integer number primitve type. Values will be represented as Java long or {@link Long}.
+   *
+   * @see ValueType#LONG
+   */
   public static final ColumnType LONG = new ColumnType(ValueType.LONG, null, null);
+  /**
+   * Druid 64-bit double precision floating point number primitive type. Values will be represented as Java double or
+   * {@link Double}.
+   *
+   * @see ValueType#DOUBLE
+   */
   public static final ColumnType DOUBLE = new ColumnType(ValueType.DOUBLE, null, null);
+  /**
+   * Druid 32-bit single precision floating point number primitive type. Values will be represented as Java float or
+   * {@link Float}.
+   *
+   * @see ValueType#FLOAT
+   */
   public static final ColumnType FLOAT = new ColumnType(ValueType.FLOAT, null, null);
   // currently, arrays only come from expressions or aggregators
   // and there are no native float expressions (or aggs which produce float arrays)
+  /**
+   * An array of Strings. Values will reprsented as Object[]

Review Comment:
   represented (spelling)



##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a
+   * {@link java.util.List<String>}.
    */
   STRING,
   /**
    * Placeholder for arbitrary 'complex' types, which have a corresponding serializer/deserializer implementation. Note
    * that knowing a type is complex alone isn't enough information to work with it directly, and additional information
-   * in the form of a type name that is registered in the complex type registry must be available to make this type
-   * meaningful. This type is not currently supported as a grouping key for aggregations, and may not be used as an
-   * input to expression virtual columns, and might only be supported by the specific aggregators crafted to handle
-   * this complex type.
+   * in the form of a type name which must be registered in the complex type registry. This type is not currently
+   * supported as a grouping key for aggregations, and might only be supported by the specific aggregators crafted to

Review Comment:
   "might"? How about:
   
   > Complex types are not currently supported as a grouping key for aggregations. Complex types can be used as inputs to aggregators, in cases where the specific aggregator supports the specific complex type.



##########
core/src/main/java/org/apache/druid/segment/column/ColumnType.java:
##########
@@ -26,18 +26,71 @@
 
 import javax.annotation.Nullable;
 
+/**
+ * Native Druid types.
+ *
+ * @see TypeSignature
+ */
 @JsonSerialize(using = ToStringSerializer.class)
 public class ColumnType extends BaseTypeSignature<ValueType>
 {
+  /**
+   * Druid string type. Values will be represented as {@link String} or {@link java.util.List<String>} in the case
+   * of multi-value string columns. {@link ColumnType} has insufficient information to distinguish between single
+   * and multi-value strings, this requires a type 'ColumnCapabilities' which is available at a higher layer.
+   *
+   * @see ValueType#STRING
+   */
   public static final ColumnType STRING = new ColumnType(ValueType.STRING, null, null);
+
+  /**
+   * Druid 64-bit integer number primitve type. Values will be represented as Java long or {@link Long}.
+   *
+   * @see ValueType#LONG
+   */
   public static final ColumnType LONG = new ColumnType(ValueType.LONG, null, null);
+  /**
+   * Druid 64-bit double precision floating point number primitive type. Values will be represented as Java double or
+   * {@link Double}.
+   *
+   * @see ValueType#DOUBLE
+   */
   public static final ColumnType DOUBLE = new ColumnType(ValueType.DOUBLE, null, null);
+  /**
+   * Druid 32-bit single precision floating point number primitive type. Values will be represented as Java float or
+   * {@link Float}.
+   *
+   * @see ValueType#FLOAT
+   */
   public static final ColumnType FLOAT = new ColumnType(ValueType.FLOAT, null, null);
   // currently, arrays only come from expressions or aggregators
   // and there are no native float expressions (or aggs which produce float arrays)
+  /**
+   * An array of Strings. Values will reprsented as Object[]
+   * @see ValueType#ARRAY
+   * @see ValueType#STRING
+   */
   public static final ColumnType STRING_ARRAY = new ColumnType(ValueType.ARRAY, null, STRING);
+  /**
+   * An array of Longs. Values may be represented as Object[] or long[]

Review Comment:
   May? Where did the certainty go? The others are "will"!



##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a
+   * {@link java.util.List<String>}.
    */
   STRING,
   /**
    * Placeholder for arbitrary 'complex' types, which have a corresponding serializer/deserializer implementation. Note
    * that knowing a type is complex alone isn't enough information to work with it directly, and additional information
-   * in the form of a type name that is registered in the complex type registry must be available to make this type
-   * meaningful. This type is not currently supported as a grouping key for aggregations, and may not be used as an
-   * input to expression virtual columns, and might only be supported by the specific aggregators crafted to handle
-   * this complex type.
+   * in the form of a type name which must be registered in the complex type registry. This type is not currently
+   * supported as a grouping key for aggregations, and might only be supported by the specific aggregators crafted to
+   * handle this complex type. Filtering on this type with standard filters will most likely have limited support, and
+   * may match as a "null" value.
+   *
+   * These types are represented by the individual Java type associated with the complex type name as defined in the
+   * type registry.
    */
   COMPLEX,
 
   /**
-   * Placeholder for arbitrary arrays of other {@link ValueType}. This type is not currently supported as a grouping
+   * Placeholder for arbitrary arrays of other {@link ValueType}. This type has limited support as a grouping

Review Comment:
   What does "limited support" mean?



##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a

Review Comment:
   "Might"?
   
   How about: "multi-value strings appear as {@code java.util.List<String>} when necessary to represent multiple values."



##########
core/src/main/java/org/apache/druid/math/expr/Evals.java:
##########
@@ -20,21 +20,22 @@
 package org.apache.druid.math.expr;
 
 import org.apache.druid.common.config.NullHandling;
-import org.apache.druid.java.util.common.logger.Logger;
 
 import javax.annotation.Nullable;
-import java.util.Arrays;
 import java.util.List;
 
 /**
  */
 public class Evals
 {
-  private static final Logger log = new Logger(Evals.class);
-
   public static boolean isAllConstants(Expr... exprs)
   {
-    return isAllConstants(Arrays.asList(exprs));
+    for (Expr expr : exprs) {

Review Comment:
   Why duplicate the code? To avoid wrapping the array?



##########
core/src/main/java/org/apache/druid/segment/column/ColumnType.java:
##########
@@ -26,18 +26,71 @@
 
 import javax.annotation.Nullable;
 
+/**
+ * Native Druid types.
+ *
+ * @see TypeSignature
+ */
 @JsonSerialize(using = ToStringSerializer.class)
 public class ColumnType extends BaseTypeSignature<ValueType>
 {
+  /**
+   * Druid string type. Values will be represented as {@link String} or {@link java.util.List<String>} in the case
+   * of multi-value string columns. {@link ColumnType} has insufficient information to distinguish between single
+   * and multi-value strings, this requires a type 'ColumnCapabilities' which is available at a higher layer.

Review Comment:
   Link to the specific hasMultipleValues function? Or mention, if it's not available from this package.



##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a
+   * {@link java.util.List<String>}.
    */
   STRING,
   /**
    * Placeholder for arbitrary 'complex' types, which have a corresponding serializer/deserializer implementation. Note
    * that knowing a type is complex alone isn't enough information to work with it directly, and additional information
-   * in the form of a type name that is registered in the complex type registry must be available to make this type
-   * meaningful. This type is not currently supported as a grouping key for aggregations, and may not be used as an
-   * input to expression virtual columns, and might only be supported by the specific aggregators crafted to handle
-   * this complex type.
+   * in the form of a type name which must be registered in the complex type registry. This type is not currently
+   * supported as a grouping key for aggregations, and might only be supported by the specific aggregators crafted to
+   * handle this complex type. Filtering on this type with standard filters will most likely have limited support, and
+   * may match as a "null" value.
+   *
+   * These types are represented by the individual Java type associated with the complex type name as defined in the
+   * type registry.
    */
   COMPLEX,
 
   /**
-   * Placeholder for arbitrary arrays of other {@link ValueType}. This type is not currently supported as a grouping
+   * Placeholder for arbitrary arrays of other {@link ValueType}. This type has limited support as a grouping
    * key for aggregations, cannot be used as an input for numerical primitive aggregations such as sums, and may have
    * limited support as an input among complex type sketch aggregators.
+   *
+   * There are currently no native ARRAY typed columns, but they may be produced by expression virtual columns,
+   * aggregators, and post-aggregators.
+   *
+   * Arrays are typically represented as Object[], but may also be Java primitive arrays such as long[], double[], or

Review Comment:
   Love the idea of this paragraph. How do you feel about this adjustment? The idea is to firm up the language and give people clearer expectations.
   
   > Arrays are represented as Object[], long[], double[], or float[]. The preferred type is Object[], since the expression system is the main consumer of arrays, and the expression system uses Object[] internally. Some code represents arrays in other ways; in particular the groupBy engine and SQL result layer. Over time we expect these usages to migrate to Object[], long[], double[], and float[].



##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a
+   * {@link java.util.List<String>}.
    */
   STRING,
   /**
    * Placeholder for arbitrary 'complex' types, which have a corresponding serializer/deserializer implementation. Note
    * that knowing a type is complex alone isn't enough information to work with it directly, and additional information
-   * in the form of a type name that is registered in the complex type registry must be available to make this type
-   * meaningful. This type is not currently supported as a grouping key for aggregations, and may not be used as an
-   * input to expression virtual columns, and might only be supported by the specific aggregators crafted to handle
-   * this complex type.
+   * in the form of a type name which must be registered in the complex type registry. This type is not currently
+   * supported as a grouping key for aggregations, and might only be supported by the specific aggregators crafted to
+   * handle this complex type. Filtering on this type with standard filters will most likely have limited support, and

Review Comment:
   "most likely"? Are there cases where they are treated as anything other than null?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953256089


##########
core/src/main/java/org/apache/druid/math/expr/ExprEval.java:
##########
@@ -429,18 +443,15 @@ public static ExprEval ofType(@Nullable ExpressionType type, @Nullable Object va
       case STRING:
         // not all who claim to be "STRING" are always a String, prepare ourselves...
         if (value instanceof String[]) {
-          return new ArrayExprEval(ExpressionType.STRING_ARRAY, (String[]) value);
+          return new ArrayExprEval(ExpressionType.STRING_ARRAY, Arrays.stream((String[]) value).toArray());

Review Comment:
   good call, looking around there are quite a few usages of steam in array stuffs, should I try to fix them all up in this PR or do as a follow-up?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953955167


##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a
+   * {@link java.util.List<String>}.
    */
   STRING,
   /**
    * Placeholder for arbitrary 'complex' types, which have a corresponding serializer/deserializer implementation. Note
    * that knowing a type is complex alone isn't enough information to work with it directly, and additional information
-   * in the form of a type name that is registered in the complex type registry must be available to make this type
-   * meaningful. This type is not currently supported as a grouping key for aggregations, and may not be used as an
-   * input to expression virtual columns, and might only be supported by the specific aggregators crafted to handle
-   * this complex type.
+   * in the form of a type name which must be registered in the complex type registry. This type is not currently
+   * supported as a grouping key for aggregations, and might only be supported by the specific aggregators crafted to
+   * handle this complex type. Filtering on this type with standard filters will most likely have limited support, and
+   * may match as a "null" value.
+   *
+   * These types are represented by the individual Java type associated with the complex type name as defined in the
+   * type registry.
    */
   COMPLEX,
 
   /**
-   * Placeholder for arbitrary arrays of other {@link ValueType}. This type is not currently supported as a grouping
+   * Placeholder for arbitrary arrays of other {@link ValueType}. This type has limited support as a grouping

Review Comment:
   Ah that makes sense. Would be good for that to come across in the comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953947645


##########
core/src/main/java/org/apache/druid/segment/column/ValueType.java:
##########
@@ -63,23 +63,37 @@ public enum ValueType implements TypeDescriptor
    * String object type. This type may be used as a grouping key, an input to certain types of complex sketch
    * aggregators, and as an input to expression virtual columns. String types might potentially be 'multi-valued' when
    * stored in segments, and contextually at various layers of query processing, but this information is not available
-   * through this enum alone, and must be accompany this type indicator to properly handle.
+   * at this level.
+   *
+   * Strings are typically represented as {@link String}, but multi-value strings might also appear as a
+   * {@link java.util.List<String>}.
    */
   STRING,
   /**
    * Placeholder for arbitrary 'complex' types, which have a corresponding serializer/deserializer implementation. Note
    * that knowing a type is complex alone isn't enough information to work with it directly, and additional information
-   * in the form of a type name that is registered in the complex type registry must be available to make this type
-   * meaningful. This type is not currently supported as a grouping key for aggregations, and may not be used as an
-   * input to expression virtual columns, and might only be supported by the specific aggregators crafted to handle
-   * this complex type.
+   * in the form of a type name which must be registered in the complex type registry. This type is not currently
+   * supported as a grouping key for aggregations, and might only be supported by the specific aggregators crafted to
+   * handle this complex type. Filtering on this type with standard filters will most likely have limited support, and

Review Comment:
   Maybe we can distill this to: "filters generally treat complex values as null, although this is neither guaranteed nor required."



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #12914: tighten up array handling, fix bug with array_slice output type inference

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #12914:
URL: https://github.com/apache/druid/pull/12914#discussion_r953945698


##########
core/src/main/java/org/apache/druid/math/expr/ExprEval.java:
##########
@@ -429,18 +443,15 @@ public static ExprEval ofType(@Nullable ExpressionType type, @Nullable Object va
       case STRING:
         // not all who claim to be "STRING" are always a String, prepare ourselves...
         if (value instanceof String[]) {
-          return new ArrayExprEval(ExpressionType.STRING_ARRAY, (String[]) value);
+          return new ArrayExprEval(ExpressionType.STRING_ARRAY, Arrays.stream((String[]) value).toArray());

Review Comment:
   Up to you, I'm OK either way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org