You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/12/24 23:02:00 UTC

[PR] Refactor advancing token to avoid duplication, avoid borrow checker issues [datafusion-sqlparser-rs]

alamb opened a new pull request, #1618:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1618

   - Follow on to https://github.com/apache/datafusion-sqlparser-rs/pull/1587
   - Part of https://github.com/apache/datafusion-sqlparser-rs/issues/1558
   
   @davisp made some great improvements to the `Parser` API to avoid copies, but now the API for managing tokens is a bit unwieldy. Before we release a new version I would like to improve the API
   
   Speicfically some of the APIs take a `&mut self` to advance `index` but return a read only reference to the token. This means the borrow checker is overly stringent and will sometimes think the parser has a mutable borrow outstanding when it doesnt.
   
   I think we can make the APIs easier to use (and thus more likely to use) with a bit of finagling and documentation
   
   See definitions of previous/current/next on https://github.com/apache/datafusion-sqlparser-rs/pull/1617
   
   I am hoping that we can get the API in shape a bit more and then deprecate some of the older style APIs. 
   
   Changes:
   1. Introduce `advance_token()` to advance to the next token and reduce redundancy
   2. Introduce `get_current_token()`, `get_next_token()` `get_prev_token`
   2. Introduce `current_token_index`
   3. Remove `next_token_ref_with_index`  and `next_token_ref` in favor of the above
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Refactor advancing token to avoid duplication, avoid borrow checker issues [datafusion-sqlparser-rs]

Posted by "iffyio (via GitHub)" <gi...@apache.org>.
iffyio commented on code in PR #1618:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1618#discussion_r1898467500


##########
src/parser/mod.rs:
##########
@@ -3641,22 +3671,31 @@ impl<'a> Parser<'a> {
     #[must_use]
     pub fn parse_keyword(&mut self, expected: Keyword) -> bool {
         if self.peek_keyword(expected) {
-            self.next_token_ref();
+            self.advance_token();
             true
         } else {
             false
         }
     }
 
+    /// If the current token is the `expected` keyword, consume it and returns
+    ///
+    /// See [`Self::parse_keyword_token_ref`] to avoid the copy.
     #[must_use]
     pub fn parse_keyword_token(&mut self, expected: Keyword) -> Option<TokenWithSpan> {
         self.parse_keyword_token_ref(expected).cloned()
     }
 
+    /// If the current token is the `expected` keyword, consume it and returns a reference to the next token.
+    ///
+    // TODO remove this in favor of advance??

Review Comment:
   Not sure I followed the intent of the comment, did it mean potentially replacing this api with something similar called `advance` or would that have a different behavior?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Refactor advancing token to avoid duplication, avoid borrow checker issues [datafusion-sqlparser-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #1618:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1618#discussion_r1898503435


##########
src/parser/mod.rs:
##########
@@ -3641,22 +3671,31 @@ impl<'a> Parser<'a> {
     #[must_use]
     pub fn parse_keyword(&mut self, expected: Keyword) -> bool {
         if self.peek_keyword(expected) {
-            self.next_token_ref();
+            self.advance_token();
             true
         } else {
             false
         }
     }
 
+    /// If the current token is the `expected` keyword, consume it and returns
+    ///
+    /// See [`Self::parse_keyword_token_ref`] to avoid the copy.
     #[must_use]
     pub fn parse_keyword_token(&mut self, expected: Keyword) -> Option<TokenWithSpan> {
         self.parse_keyword_token_ref(expected).cloned()
     }
 
+    /// If the current token is the `expected` keyword, consume it and returns a reference to the next token.
+    ///
+    // TODO remove this in favor of advance??

Review Comment:
   That is a good question -- yes I think I meant changing this API somehow so it doesn't end up preserving a mutable borrow to return the option. 
   
   I will see if I can find a better way
   
   It is probably better to remove the comment though for now to avoid leaving a mess in the codebase:
    ```suggestion
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Refactor advancing token to avoid duplication, avoid borrow checker issues [datafusion-sqlparser-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #1618:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1618#discussion_r1896991320


##########
src/parser/mod.rs:
##########
@@ -3641,22 +3670,31 @@ impl<'a> Parser<'a> {
     #[must_use]
     pub fn parse_keyword(&mut self, expected: Keyword) -> bool {
         if self.peek_keyword(expected) {
-            self.next_token_ref();
+            self.advance_token();

Review Comment:
   I think this is nicer too as it makes clear the token is unused (it is just being advanced)



##########
src/parser/mod.rs:
##########
@@ -1276,7 +1276,9 @@ impl<'a> Parser<'a> {
 
         let dialect = self.dialect;
 
-        let (next_token, next_token_index) = self.next_token_ref_with_index();
+        self.advance_token();

Review Comment:
   This has two benefits: 
   1. I think it is more explicit
   2. It doesn't not result in a `mut` borrow of `self`(this is the key difference)



##########
src/parser/mod.rs:
##########
@@ -3557,45 +3564,67 @@ impl<'a> Parser<'a> {
     ///
     /// See [`Self::next_token_ref`] to avoid the copy.
     pub fn next_token(&mut self) -> TokenWithSpan {
-        self.next_token_ref().clone()
+        self.advance_token();
+        self.get_current_token().clone()
     }
 
-    pub fn next_token_ref(&mut self) -> &TokenWithSpan {
-        self.next_token_ref_with_index().0
+    /// Returns the index of the current token
+    ///
+    /// This can be used with APIs that expect an index, such as
+    /// [`Self::token_at`]
+    pub fn get_current_index(&self) -> usize {
+        self.index.saturating_sub(1)
     }
 
-    /// Return the first non-whitespace token that has not yet been processed
-    /// and that tokens index and advances the tokens
+    /// Return the next unprocessed token, possibly whitespace.
+    pub fn next_token_no_skip(&mut self) -> Option<&TokenWithSpan> {
+        self.index += 1;
+        self.tokens.get(self.index - 1)
+    }
+
+    /// Advances the current token to the next non-whitespace token
     ///
-    /// # Notes:
-    /// OK to call repeatedly after reaching EOF.
-    pub fn next_token_ref_with_index(&mut self) -> (&TokenWithSpan, usize) {

Review Comment:
   Note thse APIs were added in https://github.com/apache/datafusion-sqlparser-rs/pull/1587 (so not yet released) so this isn't a breaking API change
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Refactor advancing token to avoid duplication, avoid borrow checker issues [datafusion-sqlparser-rs]

Posted by "iffyio (via GitHub)" <gi...@apache.org>.
iffyio commented on code in PR #1618:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1618#discussion_r1898536231


##########
src/parser/mod.rs:
##########
@@ -3641,22 +3671,31 @@ impl<'a> Parser<'a> {
     #[must_use]
     pub fn parse_keyword(&mut self, expected: Keyword) -> bool {
         if self.peek_keyword(expected) {
-            self.next_token_ref();
+            self.advance_token();
             true
         } else {
             false
         }
     }
 
+    /// If the current token is the `expected` keyword, consume it and returns
+    ///
+    /// See [`Self::parse_keyword_token_ref`] to avoid the copy.
     #[must_use]
     pub fn parse_keyword_token(&mut self, expected: Keyword) -> Option<TokenWithSpan> {
         self.parse_keyword_token_ref(expected).cloned()
     }
 
+    /// If the current token is the `expected` keyword, consume it and returns a reference to the next token.
+    ///
+    // TODO remove this in favor of advance??

Review Comment:
   Ah I see that sounds good!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Refactor advancing token to avoid duplication, avoid borrow checker issues [datafusion-sqlparser-rs]

Posted by "iffyio (via GitHub)" <gi...@apache.org>.
iffyio merged PR #1618:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1618


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Refactor advancing token to avoid duplication, avoid borrow checker issues [datafusion-sqlparser-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #1618:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1618#issuecomment-2564333072

   > Not sure I followed the intent of the comment, did it mean potentially replacing this api with something similar called advance or would that have a different behavior?
   
   > That is a good question -- yes I think I meant changing this API somehow so it doesn't end up preserving a mutable borrow to return the option.
   
   BTW I spent some time last night and I found indeed it is possible to avoid these APIs. Here is what I came up with:
   - https://github.com/apache/datafusion-sqlparser-rs/pull/1626#


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org