You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "shuvamk (via GitHub)" <gi...@apache.org> on 2026/08/04 16:59:52 UTC
[PR] Databricks: support CREATE TABLE USING, MAP columns and LONG as BIGINT [datafusion-sqlparser-rs]
shuvamk opened a new pull request, #2425:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2425
Databricks SQL is built on Spark SQL, but `DatabricksDialect` never got three of the flags `SparkSqlDialect` sets. These fail on `DatabricksDialect` and parse on `SparkSqlDialect`:
```sql
CREATE TABLE t (id BIGINT) USING DELTA
CREATE TABLE t (m MAP<STRING, INT>)
```
with `Expected: end of statement, found: USING at Line: 1, Column: 28` and `Expected: ',' or ')' after column definition, found: < at Line: 1, Column: 22`.
`CREATE TABLE t (id LONG)` parses, but builds a `DataType::Custom` holding `LONG` instead of `DataType::BigInt(None)`. Databricks gives the type as `{ BIGINT | LONG }` — https://docs.databricks.com/aws/en/sql/language-manual/data-types/bigint-type
The fix sets those three flags on `DatabricksDialect`, mirroring `src/dialect/spark.rs`. No parser change, no other dialect affected.
Tests are in `tests/sqlparser_databricks.rs` and all three fail with `src/dialect/databricks.rs` reverted. The `AGENTS.md` pre-commit checks are clean locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Databricks: support CREATE TABLE USING, MAP, LONG, DIV and the pipe operator [datafusion-sqlparser-rs]
Posted by "shuvamk (via GitHub)" <gi...@apache.org>.
shuvamk commented on PR #2425:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2425#issuecomment-5225766112
Thanks — all four addressed in the follow-up commit.
`DIV` and `|>` are both documented for Databricks SQL, so I added them: `supports_pipe_operator`, and `parse_infix` delegating to Spark, which handles `DIV` and nothing else.
On delegating the flags: I converted the ones Spark and Databricks share, but left `supports_numeric_prefix`, `supports_table_versioning` and `supports_optimize_table` as they were — Spark doesn't override those, so delegating would take them from `true` to the trait default `false` and stop `SELECT 59901_user`, `VERSION AS OF` and `OPTIMIZE` from parsing on Databricks. Happy to convert the remaining shared ones too if you'd like the whole block consistent.
The cross-dialect asserts moved to `tests/sqlparser_common.rs` so Databricks is covered, with the flag-off negatives alongside them. `MAP<K, V>` needed a per-dialect loop rather than one `is_err()` — the ungated dialects split into two different error messages.
Title and description updated for the added scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Databricks: support CREATE TABLE USING, MAP, LONG, DIV and the pipe operator [datafusion-sqlparser-rs]
Posted by "LucaCappelletti94 (via GitHub)" <gi...@apache.org>.
LucaCappelletti94 commented on code in PR #2425:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2425#discussion_r4099053123
##########
tests/sqlparser_databricks.rs:
##########
@@ -757,3 +757,62 @@ fn parse_databricks_query_entry_points() {
databricks()
.verified_stmt("CREATE VIEW filtered AS FROM main.raw.source |> WHERE id > 0 |> SELECT id");
}
+
+#[test]
+fn parse_create_table_using() {
+ match databricks().verified_stmt("CREATE TABLE t (id BIGINT) USING DELTA") {
+ Statement::CreateTable(CreateTable { hive_formats, .. }) => {
+ assert_eq!(
+ hive_formats.unwrap().storage,
+ Some(HiveIOFormat::Using {
+ format: Ident::new("DELTA")
+ })
+ );
+ }
+ s => panic!("Unexpected statement: {s:?}"),
+ }
+
+ databricks().verified_stmt("CREATE TABLE IF NOT EXISTS t (id BIGINT) USING PARQUET");
+}
+
+#[test]
+fn parse_create_table_map_type() {
+ match databricks().verified_stmt("CREATE TABLE t (m MAP<STRING, INT>)") {
+ Statement::CreateTable(CreateTable { columns, .. }) => {
+ assert_eq!(
+ columns[0].data_type,
+ DataType::Map(
+ Box::new(DataType::String(None)),
+ Box::new(DataType::Int(None)),
+ MapBracketKind::AngleBrackets
+ )
+ );
+ }
+ s => panic!("Unexpected statement: {s:?}"),
+ }
+
+ databricks().verified_stmt("CREATE TABLE t (m MAP<STRING, ARRAY<INT>>)");
+}
+
+#[test]
+fn parse_long_type_as_bigint() {
+ match databricks()
+ .one_statement_parses_to("CREATE TABLE t (id LONG)", "CREATE TABLE t (id BIGINT)")
+ {
+ Statement::CreateTable(CreateTable { columns, .. }) => {
+ assert_eq!(columns[0].data_type, DataType::BigInt(None));
+ }
+ s => panic!("Unexpected statement: {s:?}"),
+ }
+}
+
+#[test]
+fn parse_div_operator() {
+ databricks().one_statement_parses_to("SELECT 10 div 3", "SELECT 10 DIV 3");
+ databricks().one_statement_parses_to("SELECT c1 div c2 FROM t", "SELECT c1 DIV c2 FROM t");
+}
+
+#[test]
+fn parse_pipe_operator() {
+ databricks().verified_stmt("SELECT * FROM t |> WHERE x > 1 |> SELECT x AS y |> ORDER BY y");
+}
Review Comment:
You should drop `parse_pipe_operator`, as Databricks already enables `|>` on main (#2478).
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Databricks: support CREATE TABLE USING, MAP columns and LONG as BIGINT [datafusion-sqlparser-rs]
Posted by "LucaCappelletti94 (via GitHub)" <gi...@apache.org>.
LucaCappelletti94 commented on code in PR #2425:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2425#discussion_r3718899365
##########
src/dialect/databricks.rs:
##########
Review Comment:
I believe there is some argument to be made that, since several of these properties need to be kept aligned between Spark and Databricks, it may be desirable to call directly Spark methods in these methods instead of duplicating the scalar value.
##########
tests/sqlparser_databricks.rs:
##########
@@ -737,3 +737,55 @@ fn parse_cte_without_as() {
.parse_sql_statements("WITH cte (SELECT 1) SELECT * FROM cte")
.is_err());
}
+
+#[test]
+fn parse_create_table_using() {
+ match databricks().verified_stmt("CREATE TABLE t (id BIGINT) USING DELTA") {
+ Statement::CreateTable(CreateTable { hive_formats, .. }) => {
+ assert_eq!(
+ hive_formats.unwrap().storage,
+ Some(HiveIOFormat::Using {
+ format: Ident::new("DELTA")
+ })
+ );
+ }
+ s => panic!("Unexpected statement: {s:?}"),
+ }
+
+ databricks().verified_stmt("CREATE TABLE IF NOT EXISTS t (id BIGINT) USING PARQUET");
+
+ assert!(all_dialects_where(|d| !d.supports_create_table_using())
Review Comment:
Consider adding the analogous assert also for the other flags you have set
##########
tests/sqlparser_databricks.rs:
##########
@@ -737,3 +737,55 @@ fn parse_cte_without_as() {
.parse_sql_statements("WITH cte (SELECT 1) SELECT * FROM cte")
.is_err());
}
+
+#[test]
+fn parse_create_table_using() {
+ match databricks().verified_stmt("CREATE TABLE t (id BIGINT) USING DELTA") {
+ Statement::CreateTable(CreateTable { hive_formats, .. }) => {
+ assert_eq!(
+ hive_formats.unwrap().storage,
+ Some(HiveIOFormat::Using {
+ format: Ident::new("DELTA")
+ })
+ );
+ }
+ s => panic!("Unexpected statement: {s:?}"),
+ }
+
+ databricks().verified_stmt("CREATE TABLE IF NOT EXISTS t (id BIGINT) USING PARQUET");
+
+ assert!(all_dialects_where(|d| !d.supports_create_table_using())
Review Comment:
Also, I believe that since this assertion regards all dialects except databricks, it should not be in the databricks tests, but in common.
##########
src/dialect/databricks.rs:
##########
@@ -113,4 +113,21 @@ impl Dialect for DatabricksDialect {
fn supports_select_item_multi_column_alias(&self) -> bool {
true
}
+
+ /// See <https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-table-using>
+ fn supports_create_table_using(&self) -> bool {
+ true
+ }
+
+ /// `LONG` is an alias for `BIGINT` in Databricks SQL.
+ ///
+ /// See <https://docs.databricks.com/aws/en/sql/language-manual/data-types/bigint-type>
+ fn supports_long_type_as_bigint(&self) -> bool {
+ true
+ }
+
Review Comment:
While you are adding support for Spark flags in Databricks, I believe you should also add `supports_pipe_operator` and `parse_infix` for `DIV`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Databricks: support CREATE TABLE USING, MAP, LONG, DIV and the pipe operator [datafusion-sqlparser-rs]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2425:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2425#issuecomment-5823041775
## [Codecov](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2425?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 81.05%. Comparing base ([`88ffee6`](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/commit/88ffee641a2192f17b44fec658463e2963233ec3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8399c78`](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/commit/8399c786dd4e603fd14c9ce53aa1bc12b3decd28?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2425 +/- ##
==========================================
+ Coverage 81.02% 81.05% +0.03%
==========================================
Files 42 42
Lines 33436 33450 +14
Branches 33436 33450 +14
==========================================
+ Hits 27090 27112 +22
+ Misses 2789 2781 -8
Partials 3557 3557
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2425?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org