You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dtenedor (via GitHub)" <gi...@apache.org> on 2025/03/14 21:04:48 UTC
[PR] [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token [spark]
dtenedor opened a new pull request, #50284:
URL: https://github.com/apache/spark/pull/50284
### What changes were proposed in this pull request?
This PR supports `|` as an alternative to `|>` for the SQL pipe operator token.
For example, this is now supported:
```
from t
| select x, y;
```
as an alternative to this:
```
from t
|> select x, y;
```
The change is controlled by a new Spark configuration `SQLConf.SINGLE_CHARACTER_OPERATOR_PIPE_TOKEN_ENABLED` (`spark.sql.singleCharacterOperatorPipeTokenEnabled`).
### Why are the changes needed?
This is a simple and safe change and provides syntax compatibility with other languages that use `|` for this purpose such as [Splunk SPL](https://www.splunk.com/en_us/blog/learn/splunk-cheat-sheet-query-spl-regex-commands.html) and [Kusto](https://learn.microsoft.com/en-us/kusto/query/?view=microsoft-fabric).
### Does this PR introduce _any_ user-facing change?
Yes, per above. Note this change is fully backwards-compatible.
### How was this patch tested?
This PR provides unit tests and golden file tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor closed pull request #50284: [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token
URL: https://github.com/apache/spark/pull/50284
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #50284:
URL: https://github.com/apache/spark/pull/50284#issuecomment-2725767439
cc @cloud-fan @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #50284:
URL: https://github.com/apache/spark/pull/50284#issuecomment-2734747783
Yes, right.. For the record, I also red that paper, of course, while I reviewed your original SQL Pipe Syntax PR. Thank you for considering adding them as test cases.
> Making a note here to add testing for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #50284:
URL: https://github.com/apache/spark/pull/50284#issuecomment-2744416921
I looked at this some more and it seems we will have to put in some effort in the parser to differentiate this case and resolve the ambiguity. It should be possible, but will take some extra work/investigation. I'll do some experimentation, it might take some time, so I'll close this PR for now to keep it off the "open Spark PRs" list, and re-open it later when the extra testing is available, in order to help with review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #50284:
URL: https://github.com/apache/spark/pull/50284#issuecomment-2730207770
Hi @dongjoon-hyun, this is a good question. We talked with Jeff Shute, the author of the SQL pipe syntax paper from Google. He mentions that they went with `|>` because all implementing engines can support it without any issues with their parsers. Specifically, Google uses an LALR parser [1] which makes it impossible to use `|` as the token due to ambiguity with bit operations. They won't be able to add support for this.
It seems like a growing consensus in the industry is that we should all support `|>` as the primary token, but some engines may also decide to support alternative tokens in addition. This blog [2] describes the situation and mentions how another engine decided to go this direction. Please let us know your thoughts on this -- we can certainly hold off on merging this PR until we figure out together what we want the plan to be.
[1] https://github.com/google/zetasql/blob/master/bazel/bison.bzl
[2] https://superdb.org/docs/language/pipe-ambiguity/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #50284:
URL: https://github.com/apache/spark/pull/50284#issuecomment-2734474463
I found a parsing ambiguity with this syntax that the research paper mentions:
Making a note here to add testing for this. Since this example is completely ambiguous, we'll have to decide which application of the `|` token we would prefer to take precedence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #50284:
URL: https://github.com/apache/spark/pull/50284#issuecomment-2730580081
Thank you for the pointers. Yes, that's exactly what I worried.
I believe `|>' is a better choice for a long-term interoperability while the eco-system is growing. We cannot add one by one when a new system is highlighted a new token as their new feature again.
However, let me remove my review comment here for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51518][SQL] Support | as an alternative to |> for the SQL pipe operator token [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #50284:
URL: https://github.com/apache/spark/pull/50284#issuecomment-2730586193
If you don't mind, please revise the PR description with https://github.com/apache/spark/pull/50284#issuecomment-2730207770, @dtenedor . Thank you for making more progress on this area.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org