You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "EthanVieira (via GitHub)" <gi...@apache.org> on 2026/09/23 10:35:58 UTC
[PR] Databricks: support collated data types [datafusion-sqlparser-rs]
EthanVieira opened a new pull request, #2553:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2553
Adds Databricks parsing and formatting for collated data types, including collations nested inside arrays and maps.
References: https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-collation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Databricks: support collated data types [datafusion-sqlparser-rs]
Posted by "LucaCappelletti94 (via GitHub)" <gi...@apache.org>.
LucaCappelletti94 commented on code in PR #2553:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2553#discussion_r4097605552
##########
tests/sqlparser_databricks.rs:
##########
@@ -757,3 +757,10 @@ fn parse_databricks_query_entry_points() {
databricks()
.verified_stmt("CREATE VIEW filtered AS FROM main.raw.source |> WHERE id > 0 |> SELECT id");
}
+
+#[test]
+fn parse_databricks_collated_data_types() {
+ databricks().verified_stmt(
+ "CREATE TABLE t (name STRING COLLATE UTF8_BINARY, values ARRAY<STRING COLLATE UTF8_LCASE>, attrs MAP<STRING COLLATE UTF8_BINARY, STRING COLLATE UTF8_LCASE>)",
+ );
Review Comment:
Adding a red test relative to the other note.
```suggestion
);
databricks().verified_stmt("CREATE TABLE t (s STRUCT<a STRING COLLATE UTF8_LCASE>)");
databricks().verified_stmt("CREATE TABLE t (c ARRAY<ARRAY<STRING>> COLLATE UTF8_LCASE)");
assert!(TestedDialects::new(vec![Box::new(GenericDialect {})])
.parse_sql_statements("CREATE TABLE t (c ARRAY<STRING COLLATE UTF8_LCASE>)")
.is_err());
```
##########
src/parser/mod.rs:
##########
@@ -13266,6 +13275,16 @@ impl<'a> Parser<'a> {
Ok((data, trailing_bracket))
}
+ fn parse_data_type_with_optional_collation(
+ &mut self,
+ ) -> Result<(DataType, MatchedTrailingBracket), ParserError> {
+ let (mut data_type, trailing_bracket) = self.parse_data_type_helper()?;
+ if self.dialect.supports_data_type_collation() && self.parse_keyword(Keyword::COLLATE) {
Review Comment:
You should skip `COLLATE` when the inner type closed with `>>`, because that `>>` also closed the parent and the keyword belongs to an outer level. Databricks currently turns `c ARRAY<ARRAY<STRING>> COLLATE UTF8_LCASE` into `c ARRAY<ARRAY<STRING> COLLATE UTF8_LCASE>`, moving the column collation inside the array.
```suggestion
if !trailing_bracket.0
&& self.dialect.supports_data_type_collation()
&& self.parse_keyword(Keyword::COLLATE)
{
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Databricks: support collated data types [datafusion-sqlparser-rs]
Posted by "LucaCappelletti94 (via GitHub)" <gi...@apache.org>.
LucaCappelletti94 merged PR #2553:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Databricks: support collated data types [datafusion-sqlparser-rs]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2553:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2553#issuecomment-5828611765
## [Codecov](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2553?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 81.02%. Comparing base ([`9ae00e7`](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/commit/9ae00e73b54ebbca2341c3de22312a8011e98757?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8897e2f`](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/commit/8897e2f59a94656d9318d73585d1541c9c7b7d8b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 12 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2553 +/- ##
==========================================
+ Coverage 80.96% 81.02% +0.06%
==========================================
Files 42 42
Lines 33359 33455 +96
Branches 33359 33455 +96
==========================================
+ Hits 27009 27108 +99
+ Misses 2790 2789 -1
+ Partials 3560 3558 -2
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2553?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org