You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/19 14:09:57 UTC
(datafusion) branch gh-readonly-queue/main/pr-21296-fd882fbf207c52c1a512d9a53706b264c19e90ee created (now 7e1a7104b6)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-21296-fd882fbf207c52c1a512d9a53706b264c19e90ee
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 7e1a7104b6 fix(unparser): make `BigQueryDialect` more robust (#21296)
This branch includes the following new commits:
new 7e1a7104b6 fix(unparser): make `BigQueryDialect` more robust (#21296)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: fix(unparser): make `BigQueryDialect` more robust (#21296)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-21296-fd882fbf207c52c1a512d9a53706b264c19e90ee
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 7e1a7104b62d568afe3a02cc7aecb96dd265c238
Author: Sergei Grebnov <se...@gmail.com>
AuthorDate: Sun Apr 19 17:09:36 2026 +0300
fix(unparser): make `BigQueryDialect` more robust (#21296)
## Which issue does this PR close?
PR improves `BigQueryDialect` dialect to make generated SQL
`BigQuery`-compatible (fix execution errors).
## What changes are included in this PR?
Eight `Dialect` trait overrides added to `BigQueryDialect`:
https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/data-types
1. `date_field_extract_style` → `Extract` +
`scalar_function_to_sql_overrides`
BigQuery does not support `date_part()`. TPC-H Q7, Q8, Q9 fail with
`Function not found: date_part`.
| Before (error) | After |
|---|---|
| `date_part('YEAR', l_shipdate)` | `EXTRACT(YEAR FROM l_shipdate)` |
2. `interval_style` → `SQLStandard`
BigQuery does not support PostgreSQL-style interval abbreviations. TPC-H
Q4, Q20 fail with `Syntax error: Unexpected ")"`.
| Before (error) | After |
|---|---|
| `INTERVAL '3 MONS'` | `INTERVAL '3' MONTH` |
3. `float64_ast_dtype` → `Float64`
BigQuery does not support `DOUBLE`. Fails with `Type not found: DOUBLE`.
| Before (error) | After |
|---|---|
| `CAST(a AS DOUBLE)` | `CAST(a AS FLOAT64)` |
4. `supports_column_alias_in_table_alias` → `false`
BigQuery does not support column aliases in table alias definitions.
Fails with `Expected ")" but got "("`.
| Before (error) | After |
|---|---|
| `SELECT c.key FROM (...) AS c(key)` | `SELECT c.key FROM (SELECT
o_orderkey AS key FROM orders) AS c` |
5. `utf8_cast_dtype` + `large_utf8_cast_dtype` → `String`
BigQuery does not support `VARCHAR`/`TEXT`. Fails with `Type not found:
VARCHAR`, `Type not found: Text`.
| Before (error) | After |
|---|---|
| `CAST(a AS VARCHAR)` | `CAST(a AS STRING)` |
| `CAST(a AS TEXT)` | `CAST(a AS STRING)` |
6. ~`int64_cast_dtype` → `Int64`~
7. `timestamp_cast_dtype` → `Timestamp` (no timezone qualifier)
https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/data-types#timestamp_type
BigQuery does not support `TIMESTAMP WITH TIME ZONE`. Fails with `Syntax
error: Expected ')' or keyword FORMAT but got keyword WITH`. `TIMESTAMP`
should be used (preserves time zone information)/
| Before (error) | After |
|---|---|
| `CAST(a AS TIMESTAMP WITH TIME ZONE)` | `CAST(a AS TIMESTAMP)` |
## Are these changes tested?
Yes. Added `test_bigquery_dialect_overrides` unit test covering all
eight overrides, verified against BigQuery before and after.
## Are there any user-facing changes?
No API changes. `BigQueryDialect` now generates valid BigQuery SQL for
the affected expressions.
---
datafusion/sql/src/unparser/dialect.rs | 45 +++++++++++++++++++++++++++++++
datafusion/sql/src/unparser/expr.rs | 49 ++++++++++++++++++++++++++++++++++
2 files changed, 94 insertions(+)
diff --git a/datafusion/sql/src/unparser/dialect.rs b/datafusion/sql/src/unparser/dialect.rs
index 194f443749..05438a4785 100644
--- a/datafusion/sql/src/unparser/dialect.rs
+++ b/datafusion/sql/src/unparser/dialect.rs
@@ -664,6 +664,51 @@ impl Dialect for BigQueryDialect {
fn unnest_as_table_factor(&self) -> bool {
true
}
+
+ fn supports_column_alias_in_table_alias(&self) -> bool {
+ false
+ }
+
+ fn float64_ast_dtype(&self) -> ast::DataType {
+ ast::DataType::Float64
+ }
+
+ fn utf8_cast_dtype(&self) -> ast::DataType {
+ ast::DataType::String(None)
+ }
+
+ fn large_utf8_cast_dtype(&self) -> ast::DataType {
+ ast::DataType::String(None)
+ }
+
+ fn timestamp_cast_dtype(
+ &self,
+ _time_unit: &TimeUnit,
+ _tz: &Option<Arc<str>>,
+ ) -> ast::DataType {
+ ast::DataType::Timestamp(None, TimezoneInfo::None)
+ }
+
+ fn date_field_extract_style(&self) -> DateFieldExtractStyle {
+ DateFieldExtractStyle::Extract
+ }
+
+ fn interval_style(&self) -> IntervalStyle {
+ IntervalStyle::SQLStandard
+ }
+
+ fn scalar_function_to_sql_overrides(
+ &self,
+ unparser: &Unparser,
+ func_name: &str,
+ args: &[Expr],
+ ) -> Result<Option<ast::Expr>> {
+ if func_name == "date_part" {
+ return date_part_to_sql(unparser, self.date_field_extract_style(), args);
+ }
+
+ Ok(None)
+ }
}
impl BigQueryDialect {
diff --git a/datafusion/sql/src/unparser/expr.rs b/datafusion/sql/src/unparser/expr.rs
index 5954e29718..4885dde077 100644
--- a/datafusion/sql/src/unparser/expr.rs
+++ b/datafusion/sql/src/unparser/expr.rs
@@ -3526,4 +3526,53 @@ mod tests {
}
Ok(())
}
+
+ #[test]
+ fn test_bigquery_dialect_overrides() -> Result<()> {
+ let bigquery_dialect: Arc<dyn Dialect> = Arc::new(BigQueryDialect::new());
+ let unparser = Unparser::new(bigquery_dialect.as_ref());
+
+ // date_field_extract_style: EXTRACT instead of date_part
+ let expr = Expr::ScalarFunction(ScalarFunction {
+ func: Arc::new(ScalarUDF::new_from_impl(
+ datafusion_functions::datetime::date_part::DatePartFunc::new(),
+ )),
+ args: vec![lit("YEAR"), col("date_col")],
+ });
+ let actual = format!("{}", unparser.expr_to_sql(&expr)?);
+ assert_eq!(actual, "EXTRACT(YEAR FROM `date_col`)");
+
+ // interval_style: SQL standard instead of PostgresVerbose
+ let expr = interval_year_month_lit("3 months");
+ let actual = format!("{}", unparser.expr_to_sql(&expr)?);
+ assert_eq!(actual, "INTERVAL '3' MONTH");
+
+ // float64_ast_dtype: FLOAT64 instead of DOUBLE
+ let expr = cast(col("a"), DataType::Float64);
+ let actual = format!("{}", unparser.expr_to_sql(&expr)?);
+ assert_eq!(actual, "CAST(`a` AS FLOAT64)");
+
+ // supports_column_alias_in_table_alias: false
+ assert!(!bigquery_dialect.supports_column_alias_in_table_alias());
+
+ // utf8_cast_dtype: STRING instead of VARCHAR
+ let expr = cast(col("a"), DataType::Utf8);
+ let actual = format!("{}", unparser.expr_to_sql(&expr)?);
+ assert_eq!(actual, "CAST(`a` AS STRING)");
+
+ // large_utf8_cast_dtype: STRING instead of TEXT
+ let expr = cast(col("a"), DataType::LargeUtf8);
+ let actual = format!("{}", unparser.expr_to_sql(&expr)?);
+ assert_eq!(actual, "CAST(`a` AS STRING)");
+
+ // timestamp_cast_dtype: TIMESTAMP (no WITH TIME ZONE)
+ let expr = cast(
+ col("a"),
+ DataType::Timestamp(TimeUnit::Microsecond, Some("+00:00".into())),
+ );
+ let actual = format!("{}", unparser.expr_to_sql(&expr)?);
+ assert_eq!(actual, "CAST(`a` AS TIMESTAMP)");
+
+ Ok(())
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org