You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "phillipleblanc (via GitHub)" <gi...@apache.org> on 2024/11/18 03:01:28 UTC
[PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]
phillipleblanc opened a new pull request, #13462:
URL: https://github.com/apache/datafusion/pull/13462
## Which issue does this PR close?
Closes #13461
## Rationale for this change
Now that Utf8View is being returned by DataFusion, we need to ensure that when we encounter it as part of unparsing an expression, we support it properly.
## What changes are included in this PR?
Correctly maps the Utf8View Arrow data type to the existing Utf8 type for unparsing.
## Are these changes tested?
Yes
## Are there any user-facing changes?
No API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #13462:
URL: https://github.com/apache/datafusion/pull/13462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]
Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on code in PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#discussion_r1846457841
##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -2500,4 +2498,43 @@ mod tests {
}
Ok(())
}
+
+ #[test]
+ fn test_utf8_view_to_sql() -> Result<()> {
+ let dialect = CustomDialectBuilder::new()
+ .with_utf8_cast_dtype(ast::DataType::Char(None))
+ .build();
+ let unparser = Unparser::new(&dialect);
+
+ let ast_dtype = unparser.arrow_dtype_to_ast_dtype(&DataType::Utf8View)?;
+
+ assert_eq!(ast_dtype, ast::DataType::Char(None));
+
+ let expr = Expr::Cast(Cast {
+ expr: Box::new(col("a")),
+ data_type: DataType::Utf8View,
+ });
Review Comment:
```suggestion
let expr = cast(col("a"), DataType::Utf8View);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]
Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on code in PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#discussion_r1846466213
##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -2500,4 +2498,40 @@ mod tests {
}
Ok(())
}
+
+ #[test]
+ fn test_utf8_view_to_sql() -> Result<()> {
+ let dialect = CustomDialectBuilder::new()
+ .with_utf8_cast_dtype(ast::DataType::Char(None))
+ .build();
+ let unparser = Unparser::new(&dialect);
+
+ let ast_dtype = unparser.arrow_dtype_to_ast_dtype(&DataType::Utf8View)?;
+
+ assert_eq!(ast_dtype, ast::DataType::Char(None));
+
+ let expr = cast(col("a"), DataType::Utf8View);
+ let ast = unparser.expr_to_sql(&expr)?;
+
+ let actual = format!("{}", ast);
+ let expected = r#"CAST(a AS CHAR)"#.to_string();
+
+ assert_eq!(actual, expected);
+
+ let expr = Expr::Column(Column {
+ name: "a".to_string(),
+ relation: None,
+ });
+ let expr = expr.eq(Expr::Literal(ScalarValue::Utf8View(Some(
+ "hello".to_string(),
+ ))));
Review Comment:
```suggestion
let expr = expr.eq(lit(ScalarValue::Utf8View(Some("hello".to_string()))));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]
Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on code in PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#discussion_r1846457841
##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -2500,4 +2498,43 @@ mod tests {
}
Ok(())
}
+
+ #[test]
+ fn test_utf8_view_to_sql() -> Result<()> {
+ let dialect = CustomDialectBuilder::new()
+ .with_utf8_cast_dtype(ast::DataType::Char(None))
+ .build();
+ let unparser = Unparser::new(&dialect);
+
+ let ast_dtype = unparser.arrow_dtype_to_ast_dtype(&DataType::Utf8View)?;
+
+ assert_eq!(ast_dtype, ast::DataType::Char(None));
+
+ let expr = Expr::Cast(Cast {
+ expr: Box::new(col("a")),
+ data_type: DataType::Utf8View,
+ });
Review Comment:
```suggestion
let expr = cast(col("a"), DataType::Utf8View);
```
is same as `CAST(a AS Utf8View)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]
Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#issuecomment-2482857680
Maybe it will be good to add a `IS NULL` or `IS NOT NULL` test as well.
```rust
let expr = col("a").is_not_null();
let ast = unparser.expr_to_sql(&expr)?;
let actual = format!("{}", ast);
let expected = r#"a IS NOT NULL"#.to_string();
```
I also noticed that the unparser will generate an invalid `a = NULL` or `a <> NULL` if the user writes the following expression,
```rust
let expr = col("a").eq(ScalarValue::Utf8View(None));
let ast = unparser.expr_to_sql(&expr)?;
let actual = format!("{}", ast); // a = NULL
```
Not a problem as long as the user correctly uses `expr.is_null()` or `expr.is_not_null()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]
Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on code in PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#discussion_r1846485648
##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -2500,4 +2498,50 @@ mod tests {
}
Ok(())
}
+
+ #[test]
+ fn test_utf8_view_to_sql() -> Result<()> {
+ let dialect = CustomDialectBuilder::new()
+ .with_utf8_cast_dtype(ast::DataType::Char(None))
+ .build();
+ let unparser = Unparser::new(&dialect);
+
+ let ast_dtype = unparser.arrow_dtype_to_ast_dtype(&DataType::Utf8View)?;
+
+ assert_eq!(ast_dtype, ast::DataType::Char(None));
+
+ let expr = cast(col("a"), DataType::Utf8View);
+ let ast = unparser.expr_to_sql(&expr)?;
+
+ let actual = format!("{}", ast);
+ let expected = r#"CAST(a AS CHAR)"#.to_string();
+
+ assert_eq!(actual, expected);
+
+ let expr = col("a").eq(lit(ScalarValue::Utf8View(Some("hello".to_string()))));
+ let ast = unparser.expr_to_sql(&expr)?;
+
+ let actual = format!("{}", ast);
+ let expected = r#"(a = 'hello')"#.to_string();
+
+ assert_eq!(actual, expected);
+
+ let expr = col("a").is_not_null();
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]
Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on code in PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#discussion_r1846462955
##########
datafusion/sql/src/unparser/expr.rs:
##########
@@ -2500,4 +2498,43 @@ mod tests {
}
Ok(())
}
+
+ #[test]
+ fn test_utf8_view_to_sql() -> Result<()> {
+ let dialect = CustomDialectBuilder::new()
+ .with_utf8_cast_dtype(ast::DataType::Char(None))
+ .build();
+ let unparser = Unparser::new(&dialect);
+
+ let ast_dtype = unparser.arrow_dtype_to_ast_dtype(&DataType::Utf8View)?;
+
+ assert_eq!(ast_dtype, ast::DataType::Char(None));
+
+ let expr = Expr::Cast(Cast {
+ expr: Box::new(col("a")),
+ data_type: DataType::Utf8View,
+ });
+ let ast = unparser.expr_to_sql(&expr)?;
+
+ let actual = format!("{}", ast);
+ let expected = r#"CAST(a AS CHAR)"#.to_string();
+
+ assert_eq!(actual, expected);
+
+ let expr = Expr::Column(Column {
+ name: "a".to_string(),
+ relation: None,
+ });
Review Comment:
```suggestion
let expr = col("a");
```
Same is applicable here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Support Utf8View in Unparser `expr_to_sql` [datafusion]
Posted by "jcsherin (via GitHub)" <gi...@apache.org>.
jcsherin commented on PR #13462:
URL: https://github.com/apache/datafusion/pull/13462#issuecomment-2482863455
Thanks @phillipleblanc. LGTM 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org