You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "samuelcolvin (via GitHub)" <gi...@apache.org> on 2024/08/06 16:41:37 UTC

[PR] support ANY() op [datafusion]

samuelcolvin opened a new pull request, #11849:
URL: https://github.com/apache/datafusion/pull/11849

   ## Which issue does this PR close?
   
   Half of #6602 - the overwhelmingly common half.
   
   ## Rationale for this change
   
   Trying to support as much postgreSQL syntax as possible, `ANY()` queries are fairly common.
   
   ## What changes are included in this PR?
   
   * Support the `ANY()` operator, as in `'needle'=ANY(haystack)`
   * Rename the arguments to `array_has` to be less confusing
   * tests for `ANY()`
   * Some tweaks to test setup to support the above test
   
   ## Are these changes tested?
   
   Yes, I can add more if you want.
   
   ## Are there any user-facing changes?
   
   Common queries with `ANY()` should now work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] support `ANY()` op [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #11849:
URL: https://github.com/apache/datafusion/pull/11849


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] support `ANY()` op [datafusion]

Posted by "samuelcolvin (via GitHub)" <gi...@apache.org>.
samuelcolvin commented on code in PR #11849:
URL: https://github.com/apache/datafusion/pull/11849#discussion_r1706624049


##########
datafusion/expr/src/test/function_stub.rs:
##########
@@ -43,15 +43,13 @@ macro_rules! create_func {
             /// Singleton instance of [$UDAF], ensures the UDAF is only created once
             /// named STATIC_$(UDAF). For example `STATIC_FirstValue`
             #[allow(non_upper_case_globals)]
-            static [< STATIC_ $UDAF >]: std::sync::OnceLock<std::sync::Arc<crate::AggregateUDF>> =

Review Comment:
   I know, but this macro isn't public.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] support `ANY()` op [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11849:
URL: https://github.com/apache/datafusion/pull/11849#discussion_r1706047864


##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -631,6 +631,31 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
             SQLExpr::Map(map) => {
                 self.try_plan_map_literal(map.entries, schema, planner_context)
             }
+            SQLExpr::AnyOp {
+                left,
+                compare_op,
+                right,
+            } => {
+                if compare_op != BinaryOperator::Eq {
+                    return plan_err!(
+                        "Unsupported AnyOp: {compare_op}, only '=' is supported"
+                    );
+                }
+
+                let Some(fun) = self.context_provider.get_function_meta("array_has")

Review Comment:
   What do you think about making this an expr planner rather than directly getting the function via name?
   
   https://docs.rs/datafusion/latest/datafusion/logical_expr/planner/trait.ExprPlanner.html
   
   like `ExprPlanner::plan_any` or something?



##########
datafusion/functions-nested/src/array_has.rs:
##########
@@ -262,26 +262,26 @@ enum ComparisonType {
 }
 
 fn general_array_has_dispatch<O: OffsetSizeTrait>(
-    array: &ArrayRef,
-    sub_array: &ArrayRef,
+    haystack: &ArrayRef,

Review Comment:
   💯  for improved names



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] support `ANY()` op [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11849:
URL: https://github.com/apache/datafusion/pull/11849#discussion_r1706240824


##########
datafusion/expr/src/test/function_stub.rs:
##########
@@ -43,15 +43,13 @@ macro_rules! create_func {
             /// Singleton instance of [$UDAF], ensures the UDAF is only created once
             /// named STATIC_$(UDAF). For example `STATIC_FirstValue`
             #[allow(non_upper_case_globals)]
-            static [< STATIC_ $UDAF >]: std::sync::OnceLock<std::sync::Arc<crate::AggregateUDF>> =

Review Comment:
   The reason why we have full path in macro is that we can avoid repeatedly importing where ever we have macro



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] support `ANY()` op [datafusion]

Posted by "samuelcolvin (via GitHub)" <gi...@apache.org>.
samuelcolvin commented on code in PR #11849:
URL: https://github.com/apache/datafusion/pull/11849#discussion_r1706729522


##########
datafusion/functions-nested/src/planner.rs:
##########
@@ -102,7 +102,7 @@ impl ExprPlanner for NestedFunctionPlanner {
 
     fn plan_make_map(&self, args: Vec<Expr>) -> Result<PlannerResult<Vec<Expr>>> {
         if args.len() % 2 != 0 {
-            return exec_err!("make_map requires an even number of arguments");
+            return plan_err!("make_map requires an even number of arguments");

Review Comment:
   drive-by change, since this is a "plan" method I assume a "plan error" is more appropriate?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] support `ANY()` op [datafusion]

Posted by "samuelcolvin (via GitHub)" <gi...@apache.org>.
samuelcolvin commented on code in PR #11849:
URL: https://github.com/apache/datafusion/pull/11849#discussion_r1706724918


##########
datafusion/expr/src/test/function_stub.rs:
##########
@@ -43,15 +43,13 @@ macro_rules! create_func {
             /// Singleton instance of [$UDAF], ensures the UDAF is only created once
             /// named STATIC_$(UDAF). For example `STATIC_FirstValue`
             #[allow(non_upper_case_globals)]
-            static [< STATIC_ $UDAF >]: std::sync::OnceLock<std::sync::Arc<crate::AggregateUDF>> =

Review Comment:
   removed anyway.



##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -631,6 +631,31 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
             SQLExpr::Map(map) => {
                 self.try_plan_map_literal(map.entries, schema, planner_context)
             }
+            SQLExpr::AnyOp {
+                left,
+                compare_op,
+                right,
+            } => {
+                if compare_op != BinaryOperator::Eq {
+                    return plan_err!(
+                        "Unsupported AnyOp: {compare_op}, only '=' is supported"
+                    );
+                }
+
+                let Some(fun) = self.context_provider.get_function_meta("array_has")

Review Comment:
   done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org