You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "BenSatori (via GitHub)" <gi...@apache.org> on 2026/08/13 14:45:21 UTC
[PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
BenSatori opened a new pull request, #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438
The ->> (flow/pipe) operator chains multiple SQL statements in Snowflake.
**Before:** Parser rejected pipe chains like `SELECT * FROM t1 ->> SELECT * FROM $1`
**After:** Parser correctly handles:
- Chained statements with ->> operator
- Pipe result references ($1, $2, $3, etc.)
- Round-trip SQL serialization
**Example:**
```sql
SELECT * FROM users WHERE active = true ->> SELECT COUNT(*) FROM $1
```
Changes:
- Added Statement::Pipe AST variant for chained statements
- Added TableFactor::PipeResultScan for $n result references
- Extended Snowflake dialect with pipe operator support
- Comprehensive test coverage (2 test functions, multiple edge cases)
Snowflake docs:
https://docs.snowflake.com/en/sql-reference/operators-flow#pipe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
Posted by "BenSatori (via GitHub)" <gi...@apache.org>.
BenSatori commented on code in PR #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438#discussion_r3802616600
##########
src/parser/mod.rs:
##########
@@ -16706,6 +16735,18 @@ impl<'a> Parser<'a> {
// Stage reference: @mystage or @namespace.stage (e.g. Snowflake)
self.parse_snowflake_stage_table_factor()
} else {
+ // Handle Snowflake pipe result references ($1, $2, ...) in FROM clause.
+ // See <https://docs.snowflake.com/en/sql-reference/operators-flow>
+ if self.dialect.supports_snowflake_pipe_operator() {
+ if let Token::Placeholder(ref s) = self.peek_token_ref().token.clone() {
+ if let Some(index_str) = s.strip_prefix('$') {
+ if let Ok(index) = index_str.parse::<u64>() {
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
Posted by "LucaCappelletti94 (via GitHub)" <gi...@apache.org>.
LucaCappelletti94 commented on code in PR #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438#discussion_r3853701841
##########
src/parser/mod.rs:
##########
@@ -16733,6 +16763,18 @@ impl<'a> Parser<'a> {
// Stage reference: @mystage or @namespace.stage (e.g. Snowflake)
self.parse_snowflake_stage_table_factor()
} else {
+ // Handle Snowflake pipe result references ($1, $2, ...) in FROM clause.
+ // See <https://docs.snowflake.com/en/sql-reference/operators-flow>
+ if self.dialect.supports_long_arrow_pipe_operator() {
Review Comment:
I believe these could be folded? But I might be mistaken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
Posted by "BenSatori (via GitHub)" <gi...@apache.org>.
BenSatori commented on code in PR #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438#discussion_r3855040259
##########
src/parser/mod.rs:
##########
@@ -16733,6 +16763,18 @@ impl<'a> Parser<'a> {
// Stage reference: @mystage or @namespace.stage (e.g. Snowflake)
self.parse_snowflake_stage_table_factor()
} else {
+ // Handle Snowflake pipe result references ($1, $2, ...) in FROM clause.
+ // See <https://docs.snowflake.com/en/sql-reference/operators-flow>
+ if self.dialect.supports_long_arrow_pipe_operator() {
Review Comment:
Slightly folded :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438#issuecomment-5816418762
## [Codecov](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2438?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `90.00000%` with `4 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.98%. Comparing base ([`6862e71`](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/commit/6862e7180f5c0e35d7c5689d7bc5721dd9c6dca7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f8292c6`](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/commit/f8292c678998780b81af90b37b024239736619fd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2438?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [src/ast/mod.rs](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2438?src=pr&el=tree&filepath=src%2Fast%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL2FzdC9tb2QucnM=) | 71.42% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [src/ast/spans.rs](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2438?src=pr&el=tree&filepath=src%2Fast%2Fspans.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3JjL2FzdC9zcGFucy5ycw==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2438?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2438 +/- ##
==========================================
+ Coverage 80.96% 80.98% +0.01%
==========================================
Files 42 42
Lines 33386 33426 +40
Branches 33386 33426 +40
==========================================
+ Hits 27032 27069 +37
- Misses 2789 2790 +1
- Partials 3565 3567 +2
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion-sqlparser-rs/pull/2438?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
Posted by "LucaCappelletti94 (via GitHub)" <gi...@apache.org>.
LucaCappelletti94 commented on PR #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438#issuecomment-5795980771
@BenSatori GitHub CI / queue is currently chocking, it'll take a while for it to clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
Posted by "BenSatori (via GitHub)" <gi...@apache.org>.
BenSatori commented on code in PR #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438#discussion_r3802613340
##########
src/dialect/mod.rs:
##########
@@ -726,6 +726,14 @@ pub trait Dialect: Debug + Any {
false
}
+ /// Does the dialect support the Snowflake `-->` flow/pipe operator for chaining
Review Comment:
Right :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
Posted by "BenSatori (via GitHub)" <gi...@apache.org>.
BenSatori commented on code in PR #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438#discussion_r3802621667
##########
tests/sqlparser_snowflake.rs:
##########
@@ -4912,3 +4912,61 @@ fn test_select_dollar_column_from_stage() {
// With table function args, without alias
snowflake().verified_stmt("SELECT $1, $2 FROM @mystage1(file_format => 'myformat')");
}
+
+#[test]
+fn test_snowflake_pipe_operator() {
+ // Basic pipe: two SELECT statements chained
+ snowflake().verified_stmt("SELECT * FROM tablename ->> SELECT * FROM $1");
+
+ // Three statements chained
+ snowflake().verified_stmt(
+ "SELECT * FROM dept WHERE dname = 'SALES' ->> SELECT * FROM emp WHERE deptno IN (SELECT deptno FROM $1) ->> SELECT ename, sal FROM $1 ORDER BY 2 DESC",
+ );
+
+ // Reference to a non-adjacent prior result using $2
+ snowflake().verified_stmt("SELECT a FROM t ->> SELECT b FROM t2 ->> SELECT $1 FROM $2");
+
+ // Non-SELECT statements in the chain (CREATE/INSERT)
+ snowflake()
+ .verified_stmt("CREATE TABLE t (id INT) ->> INSERT INTO t VALUES (1) ->> SELECT * FROM $1");
+
+ // Pipe operator is not parsed in non-Snowflake dialects
+ use sqlparser::dialect::GenericDialect;
+ use sqlparser::parser::Parser;
+ // In a generic dialect, ->> is a binary operator, not a pipe
+ let stmts = Parser::parse_sql(&GenericDialect {}, "SELECT 1").unwrap();
Review Comment:
Added that to GenericDialect, and also renamed it so it doesn't look awkward (in case other DBs adds support for this as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
Posted by "BenSatori (via GitHub)" <gi...@apache.org>.
BenSatori commented on code in PR #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438#discussion_r3802615277
##########
tests/sqlparser_snowflake.rs:
##########
@@ -4912,3 +4912,61 @@ fn test_select_dollar_column_from_stage() {
// With table function args, without alias
snowflake().verified_stmt("SELECT $1, $2 FROM @mystage1(file_format => 'myformat')");
}
+
+#[test]
+fn test_snowflake_pipe_operator() {
Review Comment:
Added these cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
Posted by "LucaCappelletti94 (via GitHub)" <gi...@apache.org>.
LucaCappelletti94 commented on code in PR #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438#discussion_r3801456133
##########
src/dialect/mod.rs:
##########
@@ -726,6 +726,14 @@ pub trait Dialect: Debug + Any {
false
}
+ /// Does the dialect support the Snowflake `-->` flow/pipe operator for chaining
Review Comment:
This looks like a typo
##########
tests/sqlparser_snowflake.rs:
##########
@@ -4912,3 +4912,61 @@ fn test_select_dollar_column_from_stage() {
// With table function args, without alias
snowflake().verified_stmt("SELECT $1, $2 FROM @mystage1(file_format => 'myformat')");
}
+
+#[test]
+fn test_snowflake_pipe_operator() {
+ // Basic pipe: two SELECT statements chained
+ snowflake().verified_stmt("SELECT * FROM tablename ->> SELECT * FROM $1");
+
+ // Three statements chained
+ snowflake().verified_stmt(
+ "SELECT * FROM dept WHERE dname = 'SALES' ->> SELECT * FROM emp WHERE deptno IN (SELECT deptno FROM $1) ->> SELECT ename, sal FROM $1 ORDER BY 2 DESC",
+ );
+
+ // Reference to a non-adjacent prior result using $2
+ snowflake().verified_stmt("SELECT a FROM t ->> SELECT b FROM t2 ->> SELECT $1 FROM $2");
+
+ // Non-SELECT statements in the chain (CREATE/INSERT)
+ snowflake()
+ .verified_stmt("CREATE TABLE t (id INT) ->> INSERT INTO t VALUES (1) ->> SELECT * FROM $1");
+
+ // Pipe operator is not parsed in non-Snowflake dialects
+ use sqlparser::dialect::GenericDialect;
+ use sqlparser::parser::Parser;
+ // In a generic dialect, ->> is a binary operator, not a pipe
+ let stmts = Parser::parse_sql(&GenericDialect {}, "SELECT 1").unwrap();
Review Comment:
What is this even meant to test? Also, `GenericDialect` would generally be a dialect that supports all syntaxes, and should therefore likely also support this one.
##########
src/parser/mod.rs:
##########
@@ -16706,6 +16735,18 @@ impl<'a> Parser<'a> {
// Stage reference: @mystage or @namespace.stage (e.g. Snowflake)
self.parse_snowflake_stage_table_factor()
} else {
+ // Handle Snowflake pipe result references ($1, $2, ...) in FROM clause.
+ // See <https://docs.snowflake.com/en/sql-reference/operators-flow>
+ if self.dialect.supports_snowflake_pipe_operator() {
+ if let Token::Placeholder(ref s) = self.peek_token_ref().token.clone() {
+ if let Some(index_str) = s.strip_prefix('$') {
+ if let Ok(index) = index_str.parse::<u64>() {
Review Comment:
Here you are currently accepting `$0`, maybe replace with:
```
if let Ok(index @ 1..) = index_str.parse::<u64>() {
```
##########
tests/sqlparser_snowflake.rs:
##########
@@ -4912,3 +4912,61 @@ fn test_select_dollar_column_from_stage() {
// With table function args, without alias
snowflake().verified_stmt("SELECT $1, $2 FROM @mystage1(file_format => 'myformat')");
}
+
+#[test]
+fn test_snowflake_pipe_operator() {
Review Comment:
Several cases that I am not sure are currently corrected supported, such as the following ones, should be added to the tests. These are ones from the Snowflake docs:
```
// Exact documented SHOW shape.
verified_stmt(
r#"SHOW WAREHOUSES
->> SELECT "name", "state", "type", "size" FROM $1"#,
);
// A Snowflake dialect override that currently scans until EOF/semicolon.
verified_stmt("CREATE DATABASE d ->> SELECT 1");
// Existing happy path.
verified_stmt(
"CREATE TABLE t (id INT)
->> INSERT INTO t VALUES (1)
->> SELECT * FROM $1"
);
// Error paths.
assert_parse_error("SELECT 1 ->>");
assert_parse_error("SELECT * FROM $0");
// GenericDialect ambiguity.
verified_generic_expr("SELECT payload ->> 'name'");
verified_generic_pipe("SELECT 1 ->> SELECT 2");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Snowflake: Add support for ->> (pipe) operator for chaining SQL stmts [datafusion-sqlparser-rs]
Posted by "BenSatori (via GitHub)" <gi...@apache.org>.
BenSatori commented on PR #2438:
URL: https://github.com/apache/datafusion-sqlparser-rs/pull/2438#issuecomment-5795947986
@iffyio do you think we can merge this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org