You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/04/22 21:27:58 UTC

[PR] Add example of using Expr::field in `37.1.0` [datafusion]

alamb opened a new pull request, #10183:
URL: https://github.com/apache/datafusion/pull/10183

   I don't intend to merge this, but I want to put it up to illustrate how to use the 37.1.0 API to do this reasonably
   
   ## Which issue does this PR close?
   
   Part of https://github.com/apache/datafusion/issues/10181
   
   ## Rationale for this change
   
   Some users create expressions outside the context of a query that has an analyzer, etc. Some expressions now can only be used via udfs. 
   
   
   ## What changes are included in this PR?
   
   Add example of using Expr::field in `37.1.0`
   
   ## Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org


Re: [PR] Add example of using Expr::field in `37.1.0` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #10183: Add example of using Expr::field in `37.1.0`
URL: https://github.com/apache/datafusion/pull/10183


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add example of using Expr::field in `37.1.0` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10183:
URL: https://github.com/apache/datafusion/pull/10183#discussion_r1575378141


##########
datafusion-examples/examples/expr_api.rs:
##########
@@ -248,18 +296,35 @@ fn make_ts_field(name: &str) -> Field {
     make_field(name, DataType::Timestamp(TimeUnit::Nanosecond, tz))
 }
 
-/// Build a physical expression from a logical one, after applying simplification and type coercion
+/// Build a physical expression from a logical one, after applying
+/// simplification,  type coercion, and applying function rewrites
 pub fn physical_expr(schema: &Schema, expr: Expr) -> Result<Arc<dyn PhysicalExpr>> {
     let df_schema = schema.clone().to_dfschema_ref()?;
 
-    // Simplify
+    // register the standard DataFusion function library
     let props = ExecutionProps::new();
     let simplifier =
         ExprSimplifier::new(SimplifyContext::new(&props).with_schema(df_schema.clone()));
 
     // apply type coercion here to ensure types match
     let expr = simplifier.coerce(expr, df_schema.clone())?;
 
+    // Support Expr::struct by rewriting expressions
+    let expr = expr
+        .transform_up(&|expr| {
+            Ok(match expr {
+                Expr::GetIndexedField(GetIndexedField {

Review Comment:
   This basically inlines some of the code from here https://github.com/apache/datafusion/blob/95607325410d6bdd4461c66b1e6ffe01fcbd736f/datafusion/functions-array/src/rewrite.rs#L151-L157



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org


Re: [PR] Add example of using Expr::field in `37.1.0` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10183:
URL: https://github.com/apache/datafusion/pull/10183#discussion_r1575378141


##########
datafusion-examples/examples/expr_api.rs:
##########
@@ -248,18 +296,35 @@ fn make_ts_field(name: &str) -> Field {
     make_field(name, DataType::Timestamp(TimeUnit::Nanosecond, tz))
 }
 
-/// Build a physical expression from a logical one, after applying simplification and type coercion
+/// Build a physical expression from a logical one, after applying
+/// simplification,  type coercion, and applying function rewrites
 pub fn physical_expr(schema: &Schema, expr: Expr) -> Result<Arc<dyn PhysicalExpr>> {
     let df_schema = schema.clone().to_dfschema_ref()?;
 
-    // Simplify
+    // register the standard DataFusion function library
     let props = ExecutionProps::new();
     let simplifier =
         ExprSimplifier::new(SimplifyContext::new(&props).with_schema(df_schema.clone()));
 
     // apply type coercion here to ensure types match
     let expr = simplifier.coerce(expr, df_schema.clone())?;
 
+    // Support Expr::struct by rewriting expressions
+    let expr = expr
+        .transform_up(&|expr| {
+            Ok(match expr {
+                Expr::GetIndexedField(GetIndexedField {

Review Comment:
   This basically inlines the necessary code from https://github.com/apache/datafusion/blob/95607325410d6bdd4461c66b1e6ffe01fcbd736f/datafusion/functions-array/src/rewrite.rs#L151-L157



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org


Re: [PR] Add example of using Expr::field in `37.1.0` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10183:
URL: https://github.com/apache/datafusion/pull/10183#issuecomment-2096436885

   Here is the proposed public API that should complete this actual issue: https://github.com/apache/datafusion/pull/10330


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org