You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "sreemanamala (via GitHub)" <gi...@apache.org> on 2024/03/13 02:00:22 UTC

[PR] Allow non literal rhs in MV_FILTER_ONLY and MV_FILTER_NONE (druid)

sreemanamala opened a new pull request, #16113:
URL: https://github.com/apache/druid/pull/16113

   
   ### Description
   
   This PR allows to use the MV_FILTER_ONLY & MV_FILTER_NONE functions with a non literal argument. 
   Currently `select mv_filter_only('mvd_dim', 'array_dim') from 'table'` returns a `Unhandled Query Planning Failure`
   This is being tackled and also considered for the cases where the `array_dim` having null & empty values
   
   
   <hr>
   
   ##### Key changed/added classes in this PR
    * `MultiValueStringOperatorConversions`
    * `ApplyFunction`
    * `Parser`
    * `CalciteMultiValueStringQueryTest`
   
   <hr>
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
   
   This PR has:
   
   - [x] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Allow non literal rhs in MV_FILTER_ONLY and MV_FILTER_NONE (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz merged PR #16113:
URL: https://github.com/apache/druid/pull/16113


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Allow non literal rhs in MV_FILTER_ONLY and MV_FILTER_NONE (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16113:
URL: https://github.com/apache/druid/pull/16113#discussion_r1524084705


##########
processing/src/main/java/org/apache/druid/math/expr/ApplyFunction.java:
##########
@@ -476,6 +476,44 @@ private <T> Stream<T> filter(T[] array, LambdaExpr expr, SettableLambdaBinding b
     }
   }
 
+  /**
+   * Extended version of {@link FilterFunction} to return a null expr if filtered result turns out to be empty
+   */
+  class FilterOnlyFunction extends FilterFunction
+  {
+    static final String NAME = "filter_only";
+
+    @Override
+    public String name()
+    {
+      return NAME;
+    }
+
+    @Override
+    public ExprEval apply(LambdaExpr lambdaExpr, List<Expr> argsExpr, Expr.ObjectBinding bindings)
+    {
+      Expr arrayExpr = argsExpr.get(0);
+      ExprEval arrayEval = arrayExpr.eval(bindings);
+
+      Object[] array = arrayEval.asArray();
+      if (array == null) {
+        return ExprEval.of(null);
+      }
+
+      SettableLambdaBinding lambdaBinding = new SettableLambdaBinding(arrayEval.elementType(), lambdaExpr, bindings);
+      Object[] filtered = filter(arrayEval.asArray(), lambdaExpr, lambdaBinding).toArray();
+      if (filtered.length == 0) {
+        return ExprEval.of(null);
+      }
+      return ExprEval.ofArray(arrayEval.asArrayType(), filtered);
+    }
+
+    private <T> Stream<T> filter(T[] array, LambdaExpr expr, SettableLambdaBinding binding)

Review Comment:
   nit: maybe should just make this method not private in `FilterFunction`? (assuming we keep this)



##########
processing/src/main/java/org/apache/druid/math/expr/ApplyFunction.java:
##########
@@ -476,6 +476,44 @@ private <T> Stream<T> filter(T[] array, LambdaExpr expr, SettableLambdaBinding b
     }
   }
 
+  /**
+   * Extended version of {@link FilterFunction} to return a null expr if filtered result turns out to be empty

Review Comment:
   hmm, reading the docs for `filter`, https://github.com/apache/druid/blob/master/docs/querying/math-expr.md#apply-functions, it actually claims that it returns null if no elements match. Given that, I suppose we could just fix `filter` to make it match the docs instead of creating a new function. Is it actually useful to have a `filter` that returns empty array if no matches?
   
   Alternatively, if we do decide we want both behaviors, we should update those docs and add new docs for this function.



##########
processing/src/main/java/org/apache/druid/math/expr/ApplyFunction.java:
##########
@@ -476,6 +476,44 @@ private <T> Stream<T> filter(T[] array, LambdaExpr expr, SettableLambdaBinding b
     }
   }
 
+  /**
+   * Extended version of {@link FilterFunction} to return a null expr if filtered result turns out to be empty
+   */
+  class FilterOnlyFunction extends FilterFunction
+  {
+    static final String NAME = "filter_only";
+
+    @Override
+    public String name()
+    {
+      return NAME;
+    }
+
+    @Override
+    public ExprEval apply(LambdaExpr lambdaExpr, List<Expr> argsExpr, Expr.ObjectBinding bindings)
+    {
+      Expr arrayExpr = argsExpr.get(0);
+      ExprEval arrayEval = arrayExpr.eval(bindings);
+
+      Object[] array = arrayEval.asArray();
+      if (array == null) {
+        return ExprEval.of(null);
+      }
+
+      SettableLambdaBinding lambdaBinding = new SettableLambdaBinding(arrayEval.elementType(), lambdaExpr, bindings);
+      Object[] filtered = filter(arrayEval.asArray(), lambdaExpr, lambdaBinding).toArray();
+      if (filtered.length == 0) {
+        return ExprEval.of(null);

Review Comment:
   same comment about `ExprEval.ofArray(arrayEval.asArrayType(), null)`



##########
processing/src/main/java/org/apache/druid/math/expr/ApplyFunction.java:
##########
@@ -476,6 +476,44 @@ private <T> Stream<T> filter(T[] array, LambdaExpr expr, SettableLambdaBinding b
     }
   }
 
+  /**
+   * Extended version of {@link FilterFunction} to return a null expr if filtered result turns out to be empty
+   */
+  class FilterOnlyFunction extends FilterFunction
+  {
+    static final String NAME = "filter_only";
+
+    @Override
+    public String name()
+    {
+      return NAME;
+    }
+
+    @Override
+    public ExprEval apply(LambdaExpr lambdaExpr, List<Expr> argsExpr, Expr.ObjectBinding bindings)
+    {
+      Expr arrayExpr = argsExpr.get(0);
+      ExprEval arrayEval = arrayExpr.eval(bindings);
+
+      Object[] array = arrayEval.asArray();
+      if (array == null) {
+        return ExprEval.of(null);

Review Comment:
   nit: this should probably return `ExprEval.ofArray(arrayEval.asArrayType(), null)` so it doesn't change the type to STRING. This is a bug in `FilterFunction` too..



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Allow non literal rhs in MV_FILTER_ONLY and MV_FILTER_NONE (druid)

Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #16113:
URL: https://github.com/apache/druid/pull/16113#discussion_r1536306394


##########
processing/src/test/java/org/apache/druid/math/expr/ApplyFunctionTest.java:
##########
@@ -93,6 +93,10 @@ public void testFilter()
 
     assertExpr("filter((x) -> x > 2, [1, 2, 3, 4, 5])", new Long[] {3L, 4L, 5L});
     assertExpr("filter((x) -> x > 2, b)", new Long[] {3L, 4L, 5L});
+
+    String dummyNull = null;
+    assertExpr("filter((x) -> array_contains([], x), ['a', 'b'])", dummyNull);
+    assertExpr("filter((x) -> array_contains([], x), null)", dummyNull);

Review Comment:
   nit: this variable seems more chars than just using `null` directly...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org