You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alexander-beedie (via GitHub)" <gi...@apache.org> on 2025/01/15 07:21:09 UTC
[PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]
alexander-beedie opened a new pull request, #1655:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655
Supports the `IS [NOT] [form] NORMALIZED -> bool` syntax:
Details from the PostgreSQL string function docs:
https://www.postgresql.org/docs/current/functions-string.html
```
Checks whether the string is in the specified Unicode normalization
form. The optional form key word specifies the form: NFC (the default),
NFD, NFKC, or NFKD. This expression can only be used when the server
encoding is UTF8. Note that checking for normalization using this
expression is often faster than normalizing possibly already
normalized strings.
```
* NFC: Canonical Decomposition, followed by Canonical Composition.
* NFD: Canonical Decomposition.
* NFKC: Compatibility Decomposition, followed by Canonical Composition.
* NFKD: Compatibility Decomposition.
As the normalised forms are fixed (there are only these four), it seemed reasonable to return the specified one as a new `NormalizationForm` Enum (which helps the caller as they won't be responsible for normalising string case, and can jump straight into a constrained match block).
We would use this in the Polars SQL interface to add support for this syntax.
## Examples
```sql
strcol IS NORMALIZED
strcol IS NOT NORMALIZED
```
```sql
strcol IS NFKC NORMALIZED
strcol IS NOT NFKD NORMALIZED
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]
Posted by "alexander-beedie (via GitHub)" <gi...@apache.org>.
alexander-beedie commented on code in PR #1655:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655#discussion_r1918337522
##########
src/ast/value.rs:
##########
@@ -270,6 +270,27 @@ impl fmt::Display for DateTimeField {
}
}
+#[derive(Debug, Clone, PartialEq, Eq, Ord, PartialOrd, Hash)]
+#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
+#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
+pub enum NormalizationForm {
+ NFC,
Review Comment:
Good idea; done :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]
Posted by "iffyio (via GitHub)" <gi...@apache.org>.
iffyio commented on code in PR #1655:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655#discussion_r1918787980
##########
src/parser/mod.rs:
##########
@@ -8452,6 +8458,42 @@ impl<'a> Parser<'a> {
}
}
+ /// Parse a literal unicode normalization clause
+ pub fn parse_unicode_is_normalized(
+ &mut self,
+ ) -> Result<(Option<NormalizationForm>, bool), ParserError> {
+ let neg = self.parse_keyword(Keyword::NOT);
+ if self.parse_keyword(Keyword::NORMALIZED) {
+ return Ok((None, neg));
+ }
+ let index = self.index;
+ let next_token = self.next_token();
+ let normalized_form = if let Token::Word(Word {
+ value: ref s,
+ quote_style: None,
+ keyword: Keyword::NoKeyword,
+ }) = next_token.token
+ {
+ match s.to_uppercase().as_str() {
+ "NFC" => Some(NormalizationForm::NFC),
+ "NFD" => Some(NormalizationForm::NFD),
+ "NFKC" => Some(NormalizationForm::NFKC),
+ "NFKD" => Some(NormalizationForm::NFKD),
+ _ => {
+ self.index = index;
+ return self.expected("unicode normalization", next_token);
+ }
+ }
+ } else {
+ None
+ };
+ if self.parse_keyword(Keyword::NORMALIZED) {
+ return Ok((normalized_form, neg));
+ }
+ self.index = index;
Review Comment:
Thinking it might be simpler to declare the strings as keywords, also instead of manually tracking the index in the function would it be possible to use `maybe_parse`? roughly thinking something like:
```rust
let normalized_form = self.maybe_parse(|parser| {
match parser.parse_one_of_keywords(&[
Keyword::NFC, ...
]) {
Some(Keyword::NFC) = Some(NormalizationForm::NFC)
...
}
})?;
if self.parse_keyword(Keyword::NORMALIZED) {
return Ok((normalized_form, neg));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]
Posted by "alexander-beedie (via GitHub)" <gi...@apache.org>.
alexander-beedie commented on code in PR #1655:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655#discussion_r1919845417
##########
src/parser/mod.rs:
##########
@@ -8452,6 +8458,31 @@ impl<'a> Parser<'a> {
}
}
+ /// Parse a literal unicode normalization clause
+ pub fn parse_unicode_is_normalized(
+ &mut self,
+ ) -> Result<(Option<NormalizationForm>, bool), ParserError> {
Review Comment:
Sure; updated it as `-> Result<Expr, ParserError>` 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]
Posted by "alexander-beedie (via GitHub)" <gi...@apache.org>.
alexander-beedie commented on code in PR #1655:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655#discussion_r1919845417
##########
src/parser/mod.rs:
##########
@@ -8452,6 +8458,31 @@ impl<'a> Parser<'a> {
}
}
+ /// Parse a literal unicode normalization clause
+ pub fn parse_unicode_is_normalized(
+ &mut self,
+ ) -> Result<(Option<NormalizationForm>, bool), ParserError> {
Review Comment:
Sure; updated it - now returns `Result<Expr, ParserError>` 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]
Posted by "iffyio (via GitHub)" <gi...@apache.org>.
iffyio merged PR #1655:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #1655:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655#discussion_r1918318605
##########
src/ast/mod.rs:
##########
@@ -1118,7 +1124,7 @@ impl fmt::Display for LambdaFunction {
/// `OneOrManyWithParens` implements `Deref<Target = [T]>` and `IntoIterator`,
/// so you can call slice methods on it and iterate over items
/// # Examples
-/// Acessing as a slice:
+/// Accessing as a slice:
Review Comment:
Thank you for these cleanups
##########
src/ast/value.rs:
##########
@@ -270,6 +270,27 @@ impl fmt::Display for DateTimeField {
}
}
+#[derive(Debug, Clone, PartialEq, Eq, Ord, PartialOrd, Hash)]
+#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
+#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
+pub enum NormalizationForm {
+ NFC,
Review Comment:
Bonus points if you added some comments here (maybe referencing the postgres docs and defining what the `NFC` and `NFD`, etc acronyms meant
I think you could copy from your very nice PR description 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]
Posted by "alexander-beedie (via GitHub)" <gi...@apache.org>.
alexander-beedie commented on code in PR #1655:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655#discussion_r1919735863
##########
src/parser/mod.rs:
##########
@@ -8452,6 +8458,42 @@ impl<'a> Parser<'a> {
}
}
+ /// Parse a literal unicode normalization clause
+ pub fn parse_unicode_is_normalized(
+ &mut self,
+ ) -> Result<(Option<NormalizationForm>, bool), ParserError> {
+ let neg = self.parse_keyword(Keyword::NOT);
+ if self.parse_keyword(Keyword::NORMALIZED) {
+ return Ok((None, neg));
+ }
+ let index = self.index;
+ let next_token = self.next_token();
+ let normalized_form = if let Token::Word(Word {
+ value: ref s,
+ quote_style: None,
+ keyword: Keyword::NoKeyword,
+ }) = next_token.token
+ {
+ match s.to_uppercase().as_str() {
+ "NFC" => Some(NormalizationForm::NFC),
+ "NFD" => Some(NormalizationForm::NFD),
+ "NFKC" => Some(NormalizationForm::NFKC),
+ "NFKD" => Some(NormalizationForm::NFKD),
+ _ => {
+ self.index = index;
+ return self.expected("unicode normalization", next_token);
+ }
+ }
+ } else {
+ None
+ };
+ if self.parse_keyword(Keyword::NORMALIZED) {
+ return Ok((normalized_form, neg));
+ }
+ self.index = index;
Review Comment:
Nice. I hadn't previously spotted `maybe_parse` and wasn't sure if making the normalization forms keywords would be seen as "noisy", but the end result definitely seems cleaner - have updated the code accordingly ✌️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]
Posted by "alexander-beedie (via GitHub)" <gi...@apache.org>.
alexander-beedie commented on code in PR #1655:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655#discussion_r1919735863
##########
src/parser/mod.rs:
##########
@@ -8452,6 +8458,42 @@ impl<'a> Parser<'a> {
}
}
+ /// Parse a literal unicode normalization clause
+ pub fn parse_unicode_is_normalized(
+ &mut self,
+ ) -> Result<(Option<NormalizationForm>, bool), ParserError> {
+ let neg = self.parse_keyword(Keyword::NOT);
+ if self.parse_keyword(Keyword::NORMALIZED) {
+ return Ok((None, neg));
+ }
+ let index = self.index;
+ let next_token = self.next_token();
+ let normalized_form = if let Token::Word(Word {
+ value: ref s,
+ quote_style: None,
+ keyword: Keyword::NoKeyword,
+ }) = next_token.token
+ {
+ match s.to_uppercase().as_str() {
+ "NFC" => Some(NormalizationForm::NFC),
+ "NFD" => Some(NormalizationForm::NFD),
+ "NFKC" => Some(NormalizationForm::NFKC),
+ "NFKD" => Some(NormalizationForm::NFKD),
+ _ => {
+ self.index = index;
+ return self.expected("unicode normalization", next_token);
+ }
+ }
+ } else {
+ None
+ };
+ if self.parse_keyword(Keyword::NORMALIZED) {
+ return Ok((normalized_form, neg));
+ }
+ self.index = index;
Review Comment:
Nice hadn't previously spotted `maybe_parse`; I wasn't sure if making the normalization forms keywords would be seen as "noisy", but the end result definitely seems cleaner. Have updated the code accordingly ✌️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]
Posted by "iffyio (via GitHub)" <gi...@apache.org>.
iffyio commented on code in PR #1655:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655#discussion_r1919785022
##########
src/parser/mod.rs:
##########
@@ -8452,6 +8458,31 @@ impl<'a> Parser<'a> {
}
}
+ /// Parse a literal unicode normalization clause
+ pub fn parse_unicode_is_normalized(
+ &mut self,
+ ) -> Result<(Option<NormalizationForm>, bool), ParserError> {
Review Comment:
Oh the return type becomes a bit surprising by returning a bool, since this is a public API would it make sense to have it return the expr instead `Expr::IsNormalized` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org