You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "davisp (via GitHub)" <gi...@apache.org> on 2024/12/11 18:11:01 UTC
[PR] Reduce token cloning [datafusion-sqlparser-rs]
davisp opened a new pull request, #1587:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1587
This basically just takes the approach proposed by @alamb in #1561 and runs with it.
The main changes involved:
1. A number of token parsing methods introduced that return a `&TokenWithSpan` instead of a cloned `TokenWithSpan`.
2. A bunch of updates to make use of these new ref returning functions.
3. Refactoring some of the core peek/expect/consume methods to avoid unnecessary clones
4. Replace all but one use of expect_keyword with expect_keyword_is.
Results on my M1 MacBook Pro:
```
# main:00abaf218
❯ cargo bench -- --save-baseline main
Finished `bench` profile [optimized] target(s) in 0.11s
Running benches/sqlparser_bench.rs (target/release/deps/sqlparser_bench-9f7a6e6e193d8f5c)
sqlparser-rs parsing benchmark/sqlparser::select
time: [4.2036 µs 4.2197 µs 4.2372 µs]
change: [-2.4618% -2.1421% -1.8071%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
6 (6.00%) high mild
3 (3.00%) high severe
Benchmarking sqlparser-rs parsing benchmark/sqlparser::with_select: Collecting 100 samples in estimated 5.0715 s (278k iteratio
sqlparser-rs parsing benchmark/sqlparser::with_select
time: [18.360 µs 18.425 µs 18.486 µs]
change: [-5.3763% -2.5663% -0.8662%] (p = 0.02 < 0.05)
Change within noise threshold.
# reduce-token-cloning:1ffa2aa4
❯ cargo bench -- --baseline main
Compiling sqlparser v0.52.0 (/Users/davisp/github/davisp/datafusion-sqlparser-rs)
Compiling sqlparser_bench v0.1.0 (/Users/davisp/github/davisp/datafusion-sqlparser-rs/sqlparser_bench)
Finished `bench` profile [optimized] target(s) in 15.76s
Running benches/sqlparser_bench.rs (target/release/deps/sqlparser_bench-9f7a6e6e193d8f5c)
sqlparser-rs parsing benchmark/sqlparser::select
time: [2.9006 µs 2.9020 µs 2.9034 µs]
change: [-31.254% -31.013% -30.769%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
4 (4.00%) low mild
2 (2.00%) high mild
4 (4.00%) high severe
Benchmarking sqlparser-rs parsing benchmark/sqlparser::with_select: Collecting 100 samples in estimated 5.0415 s (364k iteratio
sqlparser-rs parsing benchmark/sqlparser::with_select
time: [13.785 µs 13.813 µs 13.843 µs]
change: [-25.054% -24.778% -24.494%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve parsing performance by reducing token cloning [datafusion-sqlparser-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #1587:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve parsing performance by reducing token cloning [datafusion-sqlparser-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #1587:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1587#issuecomment-2561484355
Some follow on PRs:
- https://github.com/apache/datafusion-sqlparser-rs/pull/1617
- https://github.com/apache/datafusion-sqlparser-rs/pull/1618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve parsing performance by reducing token cloning [datafusion-sqlparser-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #1587:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1587#issuecomment-2561473115
Thanks again @davisp @Dandandan and @iffyio
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve parsing performance by reducing token cloning [datafusion-sqlparser-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #1587:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1587#issuecomment-2553491628
This PR has another conflict now
What I am hoping / planning to do is make the suggested doc improvements above and get it merged in. I also think it would be fine to merge it in and make the doc comment improvements as a follow on PR
I'll try to do this if no one else beats me to it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve parsing performance by reducing token cloning [datafusion-sqlparser-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #1587:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1587#issuecomment-2561212194
I am going to merge this PR up to resolve the conflicts and make some of the suggested improvements in documentation, etc as a follow on PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Reduce token cloning [datafusion-sqlparser-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #1587:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1587#discussion_r1882362868
##########
src/parser/mod.rs:
##########
@@ -3516,37 +3578,50 @@ impl<'a> Parser<'a> {
Ok(keyword)
} else {
let keywords: Vec<String> = keywords.iter().map(|x| format!("{x:?}")).collect();
- self.expected(
+ self.expected_ref(
&format!("one of {}", keywords.join(" or ")),
- self.peek_token(),
+ self.peek_token_ref(),
)
}
}
/// If the current token is the `expected` keyword, consume the token.
/// Otherwise, return an error.
pub fn expect_keyword(&mut self, expected: Keyword) -> Result<TokenWithSpan, ParserError> {
- if let Some(token) = self.parse_keyword_token(expected) {
- Ok(token)
+ if let Some(token) = self.parse_keyword_token_ref(expected) {
+ Ok(token.clone())
+ } else {
+ self.expected_ref(format!("{:?}", &expected).as_str(), self.peek_token_ref())
+ }
+ }
+
+ /// If the current token is the `expected` keyword, consume the token.
+ /// Otherwise, return an error.
+ ///
+ /// This differs from expect_keyword only in that the matched keyword
Review Comment:
It might be really nice to add a link in the docs too:
```suggestion
/// This differs from [`Self::expect_keyword`] only in that the matched keyword
```
##########
src/parser/mod.rs:
##########
@@ -186,6 +186,15 @@ impl std::error::Error for ParserError {}
// By default, allow expressions up to this deep before erroring
const DEFAULT_REMAINING_DEPTH: usize = 50;
+// A constant EOF token that can be referenced.
Review Comment:
💯
This might be useful enough to make `pub const` as well (so users of the crate can do the same thing)
##########
src/parser/mod.rs:
##########
@@ -3516,37 +3578,50 @@ impl<'a> Parser<'a> {
Ok(keyword)
} else {
let keywords: Vec<String> = keywords.iter().map(|x| format!("{x:?}")).collect();
- self.expected(
+ self.expected_ref(
&format!("one of {}", keywords.join(" or ")),
- self.peek_token(),
+ self.peek_token_ref(),
)
}
}
/// If the current token is the `expected` keyword, consume the token.
/// Otherwise, return an error.
pub fn expect_keyword(&mut self, expected: Keyword) -> Result<TokenWithSpan, ParserError> {
Review Comment:
Maybe as a follow on PR we could mark this API as deprecated to locate other uses of this API as well as help downstream consumers upgrade 🤔
https://github.com/apache/datafusion-sqlparser-rs/blob/5de5312406fae3f69b92b12dd63c68d7fce3ed74/src/tokenizer.rs#L604
##########
src/parser/mod.rs:
##########
@@ -3376,22 +3407,26 @@ impl<'a> Parser<'a> {
matched
}
+ pub fn next_token(&mut self) -> TokenWithSpan {
Review Comment:
Can we please add doc comments to these methods as well?
Maybe we can also mention "see `next_token_ref` for a faster, non copying API"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve parsing performance by reducing token cloning [datafusion-sqlparser-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #1587:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1587#issuecomment-2543067810
I merged this PR up to main to resolve a conflict
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve parsing performance by reducing token cloning [datafusion-sqlparser-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #1587:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1587#issuecomment-2561234597
I merged up to resolve a conflict. While reviewing the code I have some ideas on how to simplify it -- I'll try and make a PR later today. It will be a fun coding exercuse
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org