You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "wiedld (via GitHub)" <gi...@apache.org> on 2024/08/31 07:35:13 UTC

[PR] Coerce BinaryView/Utf8View to LargeBinary/LargeUtf8 on output. [datafusion]

wiedld opened a new pull request, #12271:
URL: https://github.com/apache/datafusion/pull/12271

   **WIP: may add more tests.**
   
   ## Which issue does this PR close?
   
   Closes #12119 
   
   ## Rationale for this change
   
   We plan to change Datafusion to use BinaryVIew and Utf8View by default, however, the rest of the arrow ecosystem may not be ready for these data types. As such, we want to provide the option to cast the query output to a non-view type.
   
   Because either Utf8 or LargeUtf8 can represented as a Utf8View, the cast on output converts to the larger type. Same with BinaryView to LargeBinary.
   
   ## What changes are included in this PR?
   
   During type coercion, after the plan has been coerced -- then based upon the final output type determine if an additional cast is needed.
   
   
   ## Are these changes tested?
   
   Yes.
   
   ## Are there any user-facing changes?
   
   Yes. 
   I made one existing API public (not crate private), and we added the optimizer config option to cast on output.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Coerce BinaryView/Utf8View to LargeBinary/LargeUtf8 on output. [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #12271:
URL: https://github.com/apache/datafusion/pull/12271


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Coerce BinaryView/Utf8View to LargeBinary/LargeUtf8 on output. [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #12271:
URL: https://github.com/apache/datafusion/pull/12271#discussion_r1746147461


##########
datafusion/optimizer/src/analyzer/type_coercion.rs:
##########
@@ -982,6 +1056,128 @@ mod test {
         assert_analyzed_plan_eq(Arc::new(TypeCoercion::new()), plan, expected)
     }
 
+    #[test]
+    fn coerce_utf8view_output() -> Result<()> {
+        let expr = col("a");
+        let empty = empty_with_type(DataType::Utf8View);
+        let plan = LogicalPlan::Projection(Projection::try_new(
+            vec![expr.clone()],
+            Arc::clone(&empty),
+        )?);
+
+        // baseline
+        let expected = "Projection: a\n  EmptyRelation";
+        assert_analyzed_plan_eq(Arc::new(TypeCoercion::new()), plan.clone(), expected)?;
+
+        // coerce: Utf8View => LargeUtf8
+        let expect_cast = "Projection: CAST(a AS LargeUtf8)\n  EmptyRelation";
+        let mut options = ConfigOptions::default();
+        options.optimizer.expand_views_at_output = true;
+        assert_analyzed_plan_with_config_eq(
+            options,
+            Arc::new(TypeCoercion::new()),
+            plan.clone(),
+            expect_cast,
+        )?;
+
+        // request coerce --> but output in bool, not Utf8View
+        let bool_expr = col("a").lt(lit("foo"));
+        let plan_no_cast = LogicalPlan::Projection(Projection::try_new(
+            vec![bool_expr],
+            Arc::clone(&empty),
+        )?);
+        let expect_no_cast =

Review Comment:
   I found the name with `no_cast` here confusing -- this predicate does actually have a cast (just as part of the inner expression)



##########
datafusion/optimizer/src/analyzer/type_coercion.rs:
##########
@@ -514,6 +534,59 @@ impl<'a> TreeNodeRewriter for TypeCoercionRewriter<'a> {
     }
 }
 
+/// Transform a schema to use non-view types for Utf8View and BinaryView
+fn transform_schema_to_nonview(dfschema: &DFSchemaRef) -> Option<Result<DFSchema>> {
+    let metadata = dfschema.as_arrow().metadata.clone();
+    let len = dfschema.fields().len();
+    let mut transformed = false;
+
+    let (qualifiers, transformed_fields) = dfschema
+        .iter()
+        .map(|(qualifier, field)| match field.data_type() {
+            DataType::Utf8View => {
+                transformed = true;
+                (
+                    qualifier,
+                    Arc::new(Field::new(
+                        field.name(),
+                        DataType::LargeUtf8,
+                        field.is_nullable(),
+                    )),
+                )
+            }
+            DataType::BinaryView => {
+                transformed = true;
+                (
+                    qualifier,
+                    Arc::new(Field::new(
+                        field.name(),
+                        DataType::LargeBinary,
+                        field.is_nullable(),
+                    )),
+                )
+            }
+            _ => (qualifier, Arc::clone(field)),
+        })
+        .fold(
+            (Vec::with_capacity(len), Vec::with_capacity(len)),
+            |(mut qs, mut fs), (q, f)| {
+                qs.push(q.cloned());
+                fs.push(f);
+                (qs, fs)
+            },
+        );

Review Comment:
   I think this looks like `std::iterator::unzip`: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.unzip
   
   untested:
   ```suggestion
           .unzip();
   ```



##########
datafusion/optimizer/src/analyzer/type_coercion.rs:
##########
@@ -982,6 +1056,128 @@ mod test {
         assert_analyzed_plan_eq(Arc::new(TypeCoercion::new()), plan, expected)
     }
 
+    #[test]
+    fn coerce_utf8view_output() -> Result<()> {
+        let expr = col("a");
+        let empty = empty_with_type(DataType::Utf8View);
+        let plan = LogicalPlan::Projection(Projection::try_new(
+            vec![expr.clone()],
+            Arc::clone(&empty),
+        )?);
+
+        // baseline
+        let expected = "Projection: a\n  EmptyRelation";

Review Comment:
   It might be nice to create some sort of function / struct that avoids the repetition in the tests (and thus makes the intent of the test easier to understand
   
   Maybe something like
   
   ```rust
           expect_coerced_plan(expected, plan, options)
   ```



##########
datafusion/optimizer/src/analyzer/type_coercion.rs:
##########
@@ -982,6 +1056,128 @@ mod test {
         assert_analyzed_plan_eq(Arc::new(TypeCoercion::new()), plan, expected)
     }
 
+    #[test]
+    fn coerce_utf8view_output() -> Result<()> {
+        let expr = col("a");
+        let empty = empty_with_type(DataType::Utf8View);
+        let plan = LogicalPlan::Projection(Projection::try_new(
+            vec![expr.clone()],
+            Arc::clone(&empty),
+        )?);
+
+        // baseline
+        let expected = "Projection: a\n  EmptyRelation";
+        assert_analyzed_plan_eq(Arc::new(TypeCoercion::new()), plan.clone(), expected)?;
+
+        // coerce: Utf8View => LargeUtf8
+        let expect_cast = "Projection: CAST(a AS LargeUtf8)\n  EmptyRelation";
+        let mut options = ConfigOptions::default();
+        options.optimizer.expand_views_at_output = true;
+        assert_analyzed_plan_with_config_eq(
+            options,
+            Arc::new(TypeCoercion::new()),
+            plan.clone(),
+            expect_cast,
+        )?;
+
+        // request coerce --> but output in bool, not Utf8View
+        let bool_expr = col("a").lt(lit("foo"));
+        let plan_no_cast = LogicalPlan::Projection(Projection::try_new(
+            vec![bool_expr],
+            Arc::clone(&empty),
+        )?);
+        let expect_no_cast =
+            "Projection: a < CAST(Utf8(\"foo\") AS Utf8View)\n  EmptyRelation";
+        let mut options = ConfigOptions::default();
+        options.optimizer.expand_views_at_output = true;
+        assert_analyzed_plan_with_config_eq(
+            options,
+            Arc::new(TypeCoercion::new()),
+            plan_no_cast.clone(),
+            expect_no_cast,
+        )?;
+
+        // coerce start with a non-projection root logical plan node
+        let sort_expr = expr.sort(true, true);
+        let sort_plan = LogicalPlan::Sort(Sort {
+            expr: vec![sort_expr],
+            input: Arc::new(plan),
+            fetch: None,
+        });
+        let expect_cast = "Projection: CAST(a AS LargeUtf8)\n  Sort: a ASC NULLS FIRST\n    Projection: a\n      EmptyRelation";
+        let mut options = ConfigOptions::default();
+        options.optimizer.expand_views_at_output = true;
+        assert_analyzed_plan_with_config_eq(

Review Comment:
   Could you also please add a test with two levels of projections (to ensure only the top most one is  coerced)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Coerce BinaryView/Utf8View to LargeBinary/LargeUtf8 on output. [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #12271:
URL: https://github.com/apache/datafusion/pull/12271#discussion_r1746942607


##########
datafusion/optimizer/src/analyzer/type_coercion.rs:
##########
@@ -1056,124 +1052,151 @@ mod test {
         assert_analyzed_plan_eq(Arc::new(TypeCoercion::new()), plan, expected)
     }
 
+    fn coerce_on_output_if_viewtype(plan: LogicalPlan, expected: &str) -> Result<()> {
+        let mut options = ConfigOptions::default();
+        options.optimizer.expand_views_at_output = true;
+
+        assert_analyzed_plan_with_config_eq(
+            options,
+            Arc::new(TypeCoercion::new()),
+            plan.clone(),
+            expected,
+        )
+    }
+
+    fn do_not_coerce_on_output(plan: LogicalPlan, expected: &str) -> Result<()> {
+        assert_analyzed_plan_with_config_eq(
+            ConfigOptions::default(),
+            Arc::new(TypeCoercion::new()),
+            plan.clone(),
+            expected,
+        )
+    }
+
     #[test]
     fn coerce_utf8view_output() -> Result<()> {
+        // Plan A
+        // scenario: outermost utf8view projection
         let expr = col("a");
         let empty = empty_with_type(DataType::Utf8View);
         let plan = LogicalPlan::Projection(Projection::try_new(
             vec![expr.clone()],
             Arc::clone(&empty),
         )?);
-
-        // baseline
-        let expected = "Projection: a\n  EmptyRelation";
-        assert_analyzed_plan_eq(Arc::new(TypeCoercion::new()), plan.clone(), expected)?;
-
-        // coerce: Utf8View => LargeUtf8
-        let expect_cast = "Projection: CAST(a AS LargeUtf8)\n  EmptyRelation";
-        let mut options = ConfigOptions::default();
-        options.optimizer.expand_views_at_output = true;
-        assert_analyzed_plan_with_config_eq(
-            options,
-            Arc::new(TypeCoercion::new()),
-            plan.clone(),
-            expect_cast,
-        )?;
-
-        // request coerce --> but output in bool, not Utf8View
+        // Plan A: no coerce
+        let if_not_coerced = "Projection: a\n  EmptyRelation";
+        do_not_coerce_on_output(plan.clone(), if_not_coerced)?;
+        // Plan A: coerce requested: Utf8View => LargeUtf8
+        let if_coerced = "Projection: CAST(a AS LargeUtf8)\n  EmptyRelation";
+        coerce_on_output_if_viewtype(plan.clone(), if_coerced)?;
+
+        // Plan B
+        // scenario: outermost bool projection
         let bool_expr = col("a").lt(lit("foo"));
-        let plan_no_cast = LogicalPlan::Projection(Projection::try_new(
+        let bool_plan = LogicalPlan::Projection(Projection::try_new(
             vec![bool_expr],
             Arc::clone(&empty),
         )?);
-        let expect_no_cast =
+        // Plan B: no coerce
+        let if_not_coerced =
             "Projection: a < CAST(Utf8(\"foo\") AS Utf8View)\n  EmptyRelation";
-        let mut options = ConfigOptions::default();
-        options.optimizer.expand_views_at_output = true;
-        assert_analyzed_plan_with_config_eq(
-            options,
-            Arc::new(TypeCoercion::new()),
-            plan_no_cast.clone(),
-            expect_no_cast,
-        )?;
+        do_not_coerce_on_output(bool_plan.clone(), if_not_coerced)?;
+        // Plan B: coerce requested: no coercion applied
+        let if_coerced = if_not_coerced;
+        coerce_on_output_if_viewtype(bool_plan, if_coerced)?;
 
-        // coerce start with a non-projection root logical plan node
+        // Plan C
+        // scenario: with a non-projection root logical plan node
         let sort_expr = expr.sort(true, true);
         let sort_plan = LogicalPlan::Sort(Sort {
             expr: vec![sort_expr],
             input: Arc::new(plan),
             fetch: None,
         });
-        let expect_cast = "Projection: CAST(a AS LargeUtf8)\n  Sort: a ASC NULLS FIRST\n    Projection: a\n      EmptyRelation";
-        let mut options = ConfigOptions::default();
-        options.optimizer.expand_views_at_output = true;
-        assert_analyzed_plan_with_config_eq(
-            options,
-            Arc::new(TypeCoercion::new()),
-            sort_plan.clone(),
-            expect_cast,
-        )?;
+        // Plan C: no coerce
+        let if_not_coerced =
+            "Sort: a ASC NULLS FIRST\n  Projection: a\n    EmptyRelation";
+        do_not_coerce_on_output(sort_plan.clone(), if_not_coerced)?;
+        // Plan C: coerce requested: Utf8View => LargeUtf8
+        let if_coerced = "Projection: CAST(a AS LargeUtf8)\n  Sort: a ASC NULLS FIRST\n    Projection: a\n      EmptyRelation";
+        coerce_on_output_if_viewtype(sort_plan.clone(), if_coerced)?;
+
+        // Plan D
+        // scenario: two layers of projections with view types
+        let plan = LogicalPlan::Projection(Projection::try_new(
+            vec![col("a")],
+            Arc::new(sort_plan),
+        )?);
+        // Plan D: no coerce
+        let if_not_coerced = "Projection: a\n  Sort: a ASC NULLS FIRST\n    Projection: a\n      EmptyRelation";
+        do_not_coerce_on_output(plan.clone(), if_not_coerced)?;
+        // Plan B: coerce requested: Utf8View => LargeUtf8 only on outermost
+        let if_coerced = "Projection: CAST(a AS LargeUtf8)\n  Sort: a ASC NULLS FIRST\n    Projection: a\n      EmptyRelation";

Review Comment:
   ❤️ 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Coerce BinaryView/Utf8View to LargeBinary/LargeUtf8 on output. [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12271:
URL: https://github.com/apache/datafusion/pull/12271#issuecomment-2337842531

   🚀 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org