You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "phillipleblanc (via GitHub)" <gi...@apache.org> on 2024/11/18 03:01:28 UTC

[PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]

phillipleblanc opened a new pull request, #13462:
URL: https://github.com/apache/datafusion/pull/13462

   ## Which issue does this PR close?
   
   Closes #13461
   
   ## Rationale for this change
   
   Now that Utf8View is being returned by DataFusion, we need to ensure that when we encounter it as part of unparsing an expression, we support it properly.
   
   ## What changes are included in this PR?
   
   Correctly maps the Utf8View Arrow data type to the existing Utf8 type for unparsing.
   
   ## Are these changes tested?
   
   Yes
   
   ## Are there any user-facing changes?
   
   No API changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #13462:
URL: https://github.com/apache/datafusion/pull/13462


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]

Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on code in PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#discussion_r1846457841


##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -2500,4 +2498,43 @@ mod tests {
         }
         Ok(())
     }
+
+    #[test]
+    fn test_utf8_view_to_sql() -> Result<()> {
+        let dialect = CustomDialectBuilder::new()
+            .with_utf8_cast_dtype(ast::DataType::Char(None))
+            .build();
+        let unparser = Unparser::new(&dialect);
+
+        let ast_dtype = unparser.arrow_dtype_to_ast_dtype(&DataType::Utf8View)?;
+
+        assert_eq!(ast_dtype, ast::DataType::Char(None));
+
+        let expr = Expr::Cast(Cast {
+            expr: Box::new(col("a")),
+            data_type: DataType::Utf8View,
+        });

Review Comment:
   ```suggestion
           let expr = cast(col("a"), DataType::Utf8View);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]

Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on code in PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#discussion_r1846466213


##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -2500,4 +2498,40 @@ mod tests {
         }
         Ok(())
     }
+
+    #[test]
+    fn test_utf8_view_to_sql() -> Result<()> {
+        let dialect = CustomDialectBuilder::new()
+            .with_utf8_cast_dtype(ast::DataType::Char(None))
+            .build();
+        let unparser = Unparser::new(&dialect);
+
+        let ast_dtype = unparser.arrow_dtype_to_ast_dtype(&DataType::Utf8View)?;
+
+        assert_eq!(ast_dtype, ast::DataType::Char(None));
+
+        let expr = cast(col("a"), DataType::Utf8View);
+        let ast = unparser.expr_to_sql(&expr)?;
+
+        let actual = format!("{}", ast);
+        let expected = r#"CAST(a AS CHAR)"#.to_string();
+
+        assert_eq!(actual, expected);
+
+        let expr = Expr::Column(Column {
+            name: "a".to_string(),
+            relation: None,
+        });
+        let expr = expr.eq(Expr::Literal(ScalarValue::Utf8View(Some(
+            "hello".to_string(),
+        ))));

Review Comment:
   ```suggestion
           let expr = expr.eq(lit(ScalarValue::Utf8View(Some("hello".to_string()))));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]

Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on code in PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#discussion_r1846457841


##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -2500,4 +2498,43 @@ mod tests {
         }
         Ok(())
     }
+
+    #[test]
+    fn test_utf8_view_to_sql() -> Result<()> {
+        let dialect = CustomDialectBuilder::new()
+            .with_utf8_cast_dtype(ast::DataType::Char(None))
+            .build();
+        let unparser = Unparser::new(&dialect);
+
+        let ast_dtype = unparser.arrow_dtype_to_ast_dtype(&DataType::Utf8View)?;
+
+        assert_eq!(ast_dtype, ast::DataType::Char(None));
+
+        let expr = Expr::Cast(Cast {
+            expr: Box::new(col("a")),
+            data_type: DataType::Utf8View,
+        });

Review Comment:
   ```suggestion
           let expr = cast(col("a"), DataType::Utf8View);
   ```
   
   is same as `CAST(a AS Utf8View)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]

Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#issuecomment-2482857680

   Maybe it will be good to add a `IS NULL` or `IS NOT NULL` test as well.
   
   ```rust
           let expr = col("a").is_not_null();
   
           let ast = unparser.expr_to_sql(&expr)?;
           let actual = format!("{}", ast);
   
           let expected = r#"a IS NOT NULL"#.to_string();
   ```
   
   I also noticed that the unparser will generate an invalid `a = NULL` or `a <> NULL` if the user writes the following expression,
   ```rust
   let expr = col("a").eq(ScalarValue::Utf8View(None));
   let ast = unparser.expr_to_sql(&expr)?;
   
   let actual = format!("{}", ast); // a = NULL
   ```
   
   Not a problem as long as the user correctly uses `expr.is_null()` or `expr.is_not_null()`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]

Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on code in PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#discussion_r1846485648


##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -2500,4 +2498,50 @@ mod tests {
         }
         Ok(())
     }
+
+    #[test]
+    fn test_utf8_view_to_sql() -> Result<()> {
+        let dialect = CustomDialectBuilder::new()
+            .with_utf8_cast_dtype(ast::DataType::Char(None))
+            .build();
+        let unparser = Unparser::new(&dialect);
+
+        let ast_dtype = unparser.arrow_dtype_to_ast_dtype(&DataType::Utf8View)?;
+
+        assert_eq!(ast_dtype, ast::DataType::Char(None));
+
+        let expr = cast(col("a"), DataType::Utf8View);
+        let ast = unparser.expr_to_sql(&expr)?;
+
+        let actual = format!("{}", ast);
+        let expected = r#"CAST(a AS CHAR)"#.to_string();
+
+        assert_eq!(actual, expected);
+
+        let expr = col("a").eq(lit(ScalarValue::Utf8View(Some("hello".to_string()))));
+        let ast = unparser.expr_to_sql(&expr)?;
+
+        let actual = format!("{}", ast);
+        let expected = r#"(a = 'hello')"#.to_string();
+
+        assert_eq!(actual, expected);
+
+        let expr = col("a").is_not_null();

Review Comment:
   👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]

Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on code in PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#discussion_r1846462955


##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -2500,4 +2498,43 @@ mod tests {
         }
         Ok(())
     }
+
+    #[test]
+    fn test_utf8_view_to_sql() -> Result<()> {
+        let dialect = CustomDialectBuilder::new()
+            .with_utf8_cast_dtype(ast::DataType::Char(None))
+            .build();
+        let unparser = Unparser::new(&dialect);
+
+        let ast_dtype = unparser.arrow_dtype_to_ast_dtype(&DataType::Utf8View)?;
+
+        assert_eq!(ast_dtype, ast::DataType::Char(None));
+
+        let expr = Expr::Cast(Cast {
+            expr: Box::new(col("a")),
+            data_type: DataType::Utf8View,
+        });
+        let ast = unparser.expr_to_sql(&expr)?;
+
+        let actual = format!("{}", ast);
+        let expected = r#"CAST(a AS CHAR)"#.to_string();
+
+        assert_eq!(actual, expected);
+
+        let expr = Expr::Column(Column {
+            name: "a".to_string(),
+            relation: None,
+        });

Review Comment:
   ```suggestion
           let expr = col("a");
   ```
   
   Same is applicable here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]

Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#issuecomment-2482863455

   Thanks @phillipleblanc. LGTM 👍 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org