You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/06/10 09:53:21 UTC
(datafusion) 01/01: refactor: centralize SQL dialect metadata (#22840)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-22840-fb3beba58cc95cc223cc50567134611efb4bf878
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit a851dfd9d70d76848bd1e1c26fccba73ad092f39
Author: Kumar Ujjawal <uj...@gmail.com>
AuthorDate: Wed Jun 10 15:22:58 2026 +0530
refactor: centralize SQL dialect metadata (#22840)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes #123` indicates that this PR will close issue #123.
-->
- Closes #22818.
## Rationale for this change
DataFusion kept SQL dialect metadata in several separate places, which
made it easy for enum variants, canonical names, aliases, display names,
docs, and error messages to drift.
<!--
Why are you proposing this change? If this is already explained clearly
in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand
your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
- Centralize dialect metadata in one macro-driven source.
- Generate the `Dialect` enum, metadata table, canonical names, aliases,
display names, and available-dialect string from that source.
- Replace `Dialect::AVAILABLE` with `Dialect::available()`.
- Add `Dialect::metadata()` for iterating supported dialects.
- Update dialect error messages and generated config docs to use the
centralized metadata.
- Add an upgrade note for the removed public constant.
<!--
There is no need to duplicate the description in the issue here but it
is sometimes worth providing a summary of the individual changes in this
PR.
-->
## Are these changes tested?
Yes
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example,
are they covered by existing tests)?
-->
## Are there any user-facing changes?
Yes. `Dialect::AVAILABLE` is removed and replaced by
`Dialect::available()`.
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
---
datafusion-cli/src/exec.rs | 4 +-
datafusion/common/src/config.rs | 322 ++++++++++++++++-----
datafusion/core/src/execution/session_state.rs | 16 +-
.../sqllogictest/test_files/information_schema.slt | 2 +-
docs/source/library-user-guide/upgrading/55.0.0.md | 5 +
docs/source/user-guide/configs.md | 2 +-
6 files changed, 276 insertions(+), 75 deletions(-)
diff --git a/datafusion-cli/src/exec.rs b/datafusion-cli/src/exec.rs
index 800e33f645..f7d1541b93 100644
--- a/datafusion-cli/src/exec.rs
+++ b/datafusion-cli/src/exec.rs
@@ -224,7 +224,7 @@ pub(super) async fn exec_and_print(
let dialect = dialect_from_str(dialect).ok_or_else(|| {
plan_datafusion_err!(
"Unsupported SQL dialect: {dialect}. Available dialects: {}.",
- Dialect::AVAILABLE
+ Dialect::available()
)
})?;
@@ -613,7 +613,7 @@ mod tests {
let dialect = dialect_from_str(dialect).ok_or_else(|| {
plan_datafusion_err!(
"Unsupported SQL dialect: {dialect}. Available dialects: {}.",
- Dialect::AVAILABLE
+ Dialect::available()
)
})?;
for location in locations {
diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index b10761a5fe..0c26bd0841 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -278,8 +278,8 @@ config_namespace! {
/// are normalized automatically.
pub enable_options_value_normalization: bool, warn = "`enable_options_value_normalization` is deprecated and ignored", default = false
- /// Configure the SQL dialect used by DataFusion's parser; supported values include: Generic,
- /// MySQL, PostgreSQL, Hive, SQLite, Snowflake, Redshift, MsSQL, ClickHouse, BigQuery, Ansi, DuckDB, Databricks and Spark.
+ /// Configure the SQL dialect used by DataFusion's parser.
+ /// The configuration reference lists the supported values from [`Dialect::available`].
pub dialect: Dialect, default = Dialect::Generic
// no need to lowercase because `sqlparser::dialect_from_str`] is case-insensitive
@@ -323,52 +323,172 @@ config_namespace! {
}
}
-/// This is the SQL dialect used by DataFusion's parser.
-/// This mirrors [sqlparser::dialect::Dialect](https://docs.rs/sqlparser/latest/sqlparser/dialect/trait.Dialect.html)
-/// trait in order to offer an easier API and avoid adding the `sqlparser` dependency
-#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)]
-pub enum Dialect {
- #[default]
- Generic,
- MySQL,
- PostgreSQL,
- Hive,
- SQLite,
- Snowflake,
- Redshift,
- MsSQL,
- ClickHouse,
- BigQuery,
- Ansi,
- DuckDB,
- Databricks,
- Spark,
+/// Metadata for a SQL dialect supported by DataFusion configuration.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+#[non_exhaustive]
+pub struct DialectInfo {
+ pub dialect: Dialect,
+ pub canonical_name: &'static str,
+ pub display_name: &'static str,
+ pub aliases: &'static [&'static str],
+}
+
+// Keep this key in sync with the `SqlParserOptions::dialect` config path.
+const SQL_PARSER_DIALECT_CONFIG_KEY: &str = "datafusion.sql_parser.dialect";
+
+macro_rules! dialect_display_list {
+ ($($display_name:literal),+ $(,)?) => {
+ dialect_display_list!(@acc [] $($display_name),+)
+ };
+ (@acc [$($acc:tt)*] $last:literal) => {
+ concat!($($acc)* $last)
+ };
+ (@acc [$($acc:tt)*] $next:literal, $($rest:literal),+) => {
+ dialect_display_list!(@acc [$($acc)* $next, ", ",] $($rest),+)
+ };
+}
+
+macro_rules! dialect_metadata {
+ (
+ default: $default_variant:ident;
+ $(
+ $variant:ident {
+ canonical_name: $canonical_name:literal,
+ display_name: $display_name:literal,
+ aliases: [$($alias:literal),* $(,)?],
+ }
+ ),+ $(,)?
+ ) => {
+ /// This is the SQL dialect used by DataFusion's parser.
+ /// This mirrors [sqlparser::dialect::Dialect](https://docs.rs/sqlparser/latest/sqlparser/dialect/trait.Dialect.html)
+ /// trait in order to offer an easier API and avoid adding the `sqlparser` dependency
+ #[derive(Debug, Clone, Copy, PartialEq, Eq)]
+ pub enum Dialect {
+ $($variant,)+
+ }
+
+ impl Default for Dialect {
+ fn default() -> Self {
+ Self::$default_variant
+ }
+ }
+
+ const DIALECT_INFOS: &[DialectInfo] = &[
+ $(
+ DialectInfo {
+ dialect: Dialect::$variant,
+ canonical_name: $canonical_name,
+ display_name: $display_name,
+ aliases: &[$($alias),*],
+ },
+ )+
+ ];
+
+ const AVAILABLE_DIALECTS: &str = dialect_display_list!($($display_name),+);
+ const DIALECT_CONFIG_DESCRIPTION: &str = concat!(
+ "Configure the SQL dialect used by DataFusion's parser; supported values include: ",
+ dialect_display_list!($($display_name),+),
+ "."
+ );
+ };
+}
+
+dialect_metadata! {
+ default: Generic;
+ Generic {
+ canonical_name: "generic",
+ display_name: "Generic",
+ aliases: [],
+ },
+ MySQL {
+ canonical_name: "mysql",
+ display_name: "MySQL",
+ aliases: [],
+ },
+ PostgreSQL {
+ canonical_name: "postgresql",
+ display_name: "PostgreSQL",
+ aliases: ["postgres"],
+ },
+ Hive {
+ canonical_name: "hive",
+ display_name: "Hive",
+ aliases: [],
+ },
+ SQLite {
+ canonical_name: "sqlite",
+ display_name: "SQLite",
+ aliases: [],
+ },
+ Snowflake {
+ canonical_name: "snowflake",
+ display_name: "Snowflake",
+ aliases: [],
+ },
+ Redshift {
+ canonical_name: "redshift",
+ display_name: "Redshift",
+ aliases: [],
+ },
+ MsSQL {
+ canonical_name: "mssql",
+ display_name: "MsSQL",
+ aliases: [],
+ },
+ ClickHouse {
+ canonical_name: "clickhouse",
+ display_name: "ClickHouse",
+ aliases: [],
+ },
+ BigQuery {
+ canonical_name: "bigquery",
+ display_name: "BigQuery",
+ aliases: [],
+ },
+ Ansi {
+ canonical_name: "ansi",
+ display_name: "Ansi",
+ aliases: [],
+ },
+ DuckDB {
+ canonical_name: "duckdb",
+ display_name: "DuckDB",
+ aliases: [],
+ },
+ Databricks {
+ canonical_name: "databricks",
+ display_name: "Databricks",
+ aliases: [],
+ },
+ Spark {
+ canonical_name: "spark",
+ display_name: "Spark",
+ aliases: ["sparksql"],
+ },
}
impl Dialect {
- /// List of all supported dialect names, for use in error messages.
- pub const AVAILABLE: &'static str = "Generic, MySQL, PostgreSQL, Hive, SQLite, Snowflake, Redshift, \
- MsSQL, ClickHouse, BigQuery, Ansi, DuckDB, Databricks, Spark";
+ /// Return metadata for all supported dialects.
+ pub fn metadata() -> &'static [DialectInfo] {
+ DIALECT_INFOS
+ }
+
+ /// Return all supported dialect names, for use in error messages.
+ pub fn available() -> &'static str {
+ AVAILABLE_DIALECTS
+ }
+
+ fn info(&self) -> &'static DialectInfo {
+ DIALECT_INFOS
+ .iter()
+ .find(|info| info.dialect == *self)
+ .expect("all Dialect variants are listed in DIALECT_INFOS")
+ }
}
impl AsRef<str> for Dialect {
fn as_ref(&self) -> &str {
- match self {
- Self::Generic => "generic",
- Self::MySQL => "mysql",
- Self::PostgreSQL => "postgresql",
- Self::Hive => "hive",
- Self::SQLite => "sqlite",
- Self::Snowflake => "snowflake",
- Self::Redshift => "redshift",
- Self::MsSQL => "mssql",
- Self::ClickHouse => "clickhouse",
- Self::BigQuery => "bigquery",
- Self::Ansi => "ansi",
- Self::DuckDB => "duckdb",
- Self::Databricks => "databricks",
- Self::Spark => "spark",
- }
+ self.info().canonical_name
}
}
@@ -376,34 +496,31 @@ impl FromStr for Dialect {
type Err = DataFusionError;
fn from_str(s: &str) -> Result<Self, Self::Err> {
- let value = match s.to_ascii_lowercase().as_str() {
- "generic" => Self::Generic,
- "mysql" => Self::MySQL,
- "postgresql" | "postgres" => Self::PostgreSQL,
- "hive" => Self::Hive,
- "sqlite" => Self::SQLite,
- "snowflake" => Self::Snowflake,
- "redshift" => Self::Redshift,
- "mssql" => Self::MsSQL,
- "clickhouse" => Self::ClickHouse,
- "bigquery" => Self::BigQuery,
- "ansi" => Self::Ansi,
- "duckdb" => Self::DuckDB,
- "databricks" => Self::Databricks,
- "spark" | "sparksql" => Self::Spark,
- other => {
- return Err(DataFusionError::Configuration(format!(
- "Invalid Dialect: {other}. Expected one of: {}",
- Self::AVAILABLE
- )));
+ for info in DIALECT_INFOS {
+ if info.canonical_name.eq_ignore_ascii_case(s)
+ || info
+ .aliases
+ .iter()
+ .any(|alias| alias.eq_ignore_ascii_case(s))
+ {
+ return Ok(info.dialect);
}
- };
- Ok(value)
+ }
+
+ Err(DataFusionError::Configuration(format!(
+ "Invalid Dialect: {s}. Expected one of: {}",
+ Self::available()
+ )))
}
}
impl ConfigField for Dialect {
fn visit<V: Visit>(&self, v: &mut V, key: &str, description: &'static str) {
+ let description = if key == SQL_PARSER_DIALECT_CONFIG_KEY {
+ DIALECT_CONFIG_DESCRIPTION
+ } else {
+ description
+ };
v.some(key, self, description)
}
@@ -4182,15 +4299,82 @@ mod tests {
}
#[test]
- fn test_dialect_spark_roundtrip() {
+ fn test_dialect_metadata_roundtrip() {
+ use crate::config::Dialect;
+ use std::str::FromStr;
+
+ assert_eq!(Dialect::default(), Dialect::Generic);
+ assert!(!Dialect::metadata().is_empty());
+
+ for info in Dialect::metadata() {
+ let dialect = info.dialect;
+
+ assert_eq!(Dialect::from_str(info.canonical_name).unwrap(), dialect);
+ assert_eq!(
+ Dialect::from_str(&info.canonical_name.to_ascii_uppercase()).unwrap(),
+ dialect
+ );
+ assert_eq!(dialect.as_ref(), info.canonical_name);
+ assert_eq!(dialect.to_string(), info.canonical_name);
+ }
+ }
+
+ #[test]
+ fn test_dialect_aliases() {
+ use crate::config::Dialect;
+ use std::str::FromStr;
+
+ for info in Dialect::metadata() {
+ for alias in info.aliases {
+ assert_eq!(Dialect::from_str(alias).unwrap(), info.dialect);
+ assert_eq!(
+ Dialect::from_str(&alias.to_ascii_uppercase()).unwrap(),
+ info.dialect
+ );
+ }
+ }
+ }
+
+ #[test]
+ fn test_available_dialects_includes_each_display_name_once() {
+ use crate::config::Dialect;
+ use std::collections::BTreeSet;
+
+ let available = Dialect::available();
+ let listed: Vec<_> = available.split(", ").collect();
+ let display_names: Vec<_> = Dialect::metadata()
+ .iter()
+ .map(|info| info.display_name)
+ .collect();
+ let unique_display_names: BTreeSet<_> = display_names.iter().copied().collect();
+
+ assert_eq!(display_names.len(), unique_display_names.len());
+ assert_eq!(listed, display_names);
+ }
+
+ #[test]
+ fn test_dialect_config_description_uses_metadata() {
+ use crate::config::{ConfigOptions, Dialect, SQL_PARSER_DIALECT_CONFIG_KEY};
+
+ let description = ConfigOptions::default()
+ .entries()
+ .into_iter()
+ .find(|entry| entry.key == SQL_PARSER_DIALECT_CONFIG_KEY)
+ .unwrap()
+ .description;
+
+ assert!(description.contains(Dialect::available()));
+ }
+
+ #[test]
+ fn test_invalid_dialect_error_lists_available_dialects() {
use crate::config::Dialect;
use std::str::FromStr;
- assert_eq!(Dialect::from_str("spark").unwrap(), Dialect::Spark);
- assert_eq!(Dialect::from_str("sparksql").unwrap(), Dialect::Spark);
- assert_eq!(Dialect::from_str("SPARK").unwrap(), Dialect::Spark);
- assert_eq!(Dialect::Spark.as_ref(), "spark");
- assert_eq!(Dialect::Spark.to_string(), "spark");
+ let error = Dialect::from_str("notadialect").unwrap_err().to_string();
+
+ assert!(error.contains("Invalid Dialect: notadialect"));
+ assert!(error.contains(Dialect::available()));
}
#[test]
diff --git a/datafusion/core/src/execution/session_state.rs b/datafusion/core/src/execution/session_state.rs
index dfd1eea709..ad525ac7b1 100644
--- a/datafusion/core/src/execution/session_state.rs
+++ b/datafusion/core/src/execution/session_state.rs
@@ -440,7 +440,7 @@ impl SessionState {
let dialect = dialect_from_str(dialect).ok_or_else(|| {
plan_datafusion_err!(
"Unsupported SQL dialect: {dialect}. Available dialects: {}.",
- Dialect::AVAILABLE
+ Dialect::available()
)
})?;
@@ -488,7 +488,7 @@ impl SessionState {
let dialect = dialect_from_str(dialect).ok_or_else(|| {
plan_datafusion_err!(
"Unsupported SQL dialect: {dialect}. Available dialects: {}.",
- Dialect::AVAILABLE
+ Dialect::available()
)
})?;
@@ -2371,6 +2371,18 @@ mod tests {
use std::collections::HashMap;
use std::sync::Arc;
+ #[test]
+ #[cfg(feature = "sql")]
+ fn test_configured_dialect_names_are_accepted_by_sqlparser() {
+ for info in Dialect::metadata() {
+ assert!(
+ sqlparser::dialect::dialect_from_str(info.canonical_name).is_some(),
+ "sqlparser should accept configured dialect {}",
+ info.canonical_name
+ );
+ }
+ }
+
#[test]
#[cfg(feature = "sql")]
fn test_session_state_with_default_features() {
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt b/datafusion/sqllogictest/test_files/information_schema.slt
index 8d334d8433..ea1b0aefe9 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -507,7 +507,7 @@ datafusion.runtime.temp_directory NULL The path to the temporary file directory.
datafusion.spark.map_key_dedup_policy EXCEPTION Policy for handling duplicate keys in Spark-compatible map-construction functions (`map_from_arrays`, `map_from_entries`, `str_to_map`). Mirrors Spark's [`spark.sql.mapKeyDedupPolicy`](https://github.com/apache/spark/blob/cf3a34e19dfcf70e2d679217ff1ba21302212472/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4961): - `EXCEPTION` (default): raise `[DUPLICATED_MAP_KEY]` at runtime on any duplicate key. - `LAST_WIN`: [...]
datafusion.sql_parser.collect_spans false When set to true, the source locations relative to the original SQL query (i.e. [`Span`](https://docs.rs/sqlparser/latest/sqlparser/tokenizer/struct.Span.html)) will be collected and recorded in the logical plan nodes.
datafusion.sql_parser.default_null_ordering nulls_max Specifies the default null ordering for query results. There are 4 options: - `nulls_max`: Nulls appear last in ascending order. - `nulls_min`: Nulls appear first in ascending order. - `nulls_first`: Nulls always be first in any order. - `nulls_last`: Nulls always be last in any order. By default, `nulls_max` is used to follow Postgres's behavior. postgres rule: <https://www.postgresql.org/docs/current/queries-order.html>
-datafusion.sql_parser.dialect generic Configure the SQL dialect used by DataFusion's parser; supported values include: Generic, MySQL, PostgreSQL, Hive, SQLite, Snowflake, Redshift, MsSQL, ClickHouse, BigQuery, Ansi, DuckDB, Databricks and Spark.
+datafusion.sql_parser.dialect generic Configure the SQL dialect used by DataFusion's parser; supported values include: Generic, MySQL, PostgreSQL, Hive, SQLite, Snowflake, Redshift, MsSQL, ClickHouse, BigQuery, Ansi, DuckDB, Databricks, Spark.
datafusion.sql_parser.enable_ident_normalization true When set to true, SQL parser will normalize ident (convert ident to lowercase when not quoted)
datafusion.sql_parser.enable_options_value_normalization false When set to true, SQL parser will normalize options value (convert value to lowercase). Note that this option is ignored and will be removed in the future. All case-insensitive values are normalized automatically.
datafusion.sql_parser.enable_subquery_sort_elimination true When set to true, DataFusion may remove `ORDER BY` clauses from subqueries or CTEs during SQL planning when their ordering cannot affect the result, such as when no `LIMIT` or other order-sensitive operator depends on them. Disable this option to preserve explicit subquery ordering in the planned query.
diff --git a/docs/source/library-user-guide/upgrading/55.0.0.md b/docs/source/library-user-guide/upgrading/55.0.0.md
index ad50a37cb9..2ebb6952fe 100644
--- a/docs/source/library-user-guide/upgrading/55.0.0.md
+++ b/docs/source/library-user-guide/upgrading/55.0.0.md
@@ -25,6 +25,11 @@
in this section pertains to features and changes that have already been merged
to the main branch and are awaiting release in this version.
+### `Dialect::AVAILABLE` replaced by `Dialect::available()`
+
+`datafusion_common::config::Dialect::AVAILABLE` has been removed. Use
+`Dialect::available()` instead.
+
### Decimal scalar formatting uses human-readable values
Decimal scalar literals in `EXPLAIN` output, expression display strings, and
diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md
index fa9213b965..86f0f9f3a3 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -193,7 +193,7 @@ The following configuration settings are available:
| datafusion.sql_parser.parse_float_as_decimal | false | When set to true, SQL parser will parse float as decimal type [...]
| datafusion.sql_parser.enable_ident_normalization | true | When set to true, SQL parser will normalize ident (convert ident to lowercase when not quoted) [...]
| datafusion.sql_parser.enable_options_value_normalization | false | When set to true, SQL parser will normalize options value (convert value to lowercase). Note that this option is ignored and will be removed in the future. All case-insensitive values are normalized automatically. [...]
-| datafusion.sql_parser.dialect | generic | Configure the SQL dialect used by DataFusion's parser; supported values include: Generic, MySQL, PostgreSQL, Hive, SQLite, Snowflake, Redshift, MsSQL, ClickHouse, BigQuery, Ansi, DuckDB, Databricks and Spark. [...]
+| datafusion.sql_parser.dialect | generic | Configure the SQL dialect used by DataFusion's parser; supported values include: Generic, MySQL, PostgreSQL, Hive, SQLite, Snowflake, Redshift, MsSQL, ClickHouse, BigQuery, Ansi, DuckDB, Databricks, Spark. [...]
| datafusion.sql_parser.support_varchar_with_length | true | If true, permit lengths for `VARCHAR` such as `VARCHAR(20)`, but ignore the length. If false, error if a `VARCHAR` with a length is specified. The Arrow type system does not have a notion of maximum string length and thus DataFusion can not enforce such limits. [...]
| datafusion.sql_parser.map_string_types_to_utf8view | true | If true, string types (VARCHAR, CHAR, Text, and String) are mapped to `Utf8View` during SQL planning. If false, they are mapped to `Utf8`. Default is true. [...]
| datafusion.sql_parser.collect_spans | false | When set to true, the source locations relative to the original SQL query (i.e. [`Span`](https://docs.rs/sqlparser/latest/sqlparser/tokenizer/struct.Span.html)) will be collected and recorded in the logical plan nodes. [...]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org