You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "edmondop (via GitHub)" <gi...@apache.org> on 2026/10/05 11:42:55 UTC
[I] Let a Dialect override tokenization [datafusion-sqlparser-rs]
edmondop opened a new issue, #2612:
URL: https://github.com/apache/datafusion-sqlparser-rs/issues/2612
Follow-up to the discussion in #2610.
**Problem**
The parser lets a dialect take over at specific points (`parse_prefix`, `parse_infix`, `parse_statement`): it receives `&mut Parser`, and returning `None` falls back to the default. The tokenizer has no equivalent. A dialect can only answer yes/no questions the tokenizer asks (`is_identifier_part`, `supports_triple_quoted_string`, and about 20 others), so it can toggle existing lexemes but not add new ones.
Dialect-specific lexing therefore lives in `Tokenizer::next_token` behind `dialect_of!` checks:
- `b'...'` byte strings (BigQuery, MySQL) and bit strings (Postgres)
- `r'...'` raw strings (BigQuery)
- `//` as a line comment (Snowflake) or integer division (DuckDB)
- `#` as a line comment (Snowflake, BigQuery, MySQL, Hive)
- `\r` ending a line comment (Postgres)
A downstream dialect cannot add a branch like these. Its only option is `tokenize_with_location_into_buf_with_mapper`, which rewrites one finished token at a time and cannot see characters or merge tokens.
Example: in dbt models, a Jinja comment `{# ... #}` breaks under every dialect that treats `#` as a line comment, because the tokenizer consumes the rest of the line, including `#}`.
**Proposal**
1. Expose the character stream that `next_token` reads from (today the private `State`: `peek`, `next`, `location`), or a narrow public wrapper around it.
2. Add a hook with the same contract as the parser hooks:
```rust
/// Dialect-specific tokenizer override, called before the default rules.
///
/// If `None` is returned, falls back to the default behavior.
fn next_token(
&self,
_chars: &mut State,
_prev_token: Option<&Token>,
) -> Option<Result<Token, TokenizerError>> {
None
}
```
3. As a separate step, move the existing `dialect_of!` branches into the hook implementations of their dialects. That would show the hook covers syntax already in the tree, and would remove `dialect_of!` from the tokenizer.
**Open questions**
- Lookahead: a hook that returns `None` must not consume input. `State` wraps `Peekable<Chars>` and allows one character of lookahead. Should `State` be `Clone`, or offer `peek_nth`?
- `Token` is a closed enum, so a dialect-specific lexeme must map to an existing variant (`Word`, `CustomBinaryOperator`, ...). Is that enough, or should there be a `Token::Custom` variant?
- Cost: one extra dynamic call per token. The tokenizer already calls several dialect methods per token, and `sqlparser_bench` can measure the difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org