You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2020/04/13 04:54:00 UTC
[jira] [Reopened] (SPARK-31376) Non-global sort support for
structured streaming
[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon reopened SPARK-31376:
----------------------------------
> Non-global sort support for structured streaming
> ------------------------------------------------
>
> Key: SPARK-31376
> URL: https://issues.apache.org/jira/browse/SPARK-31376
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.1.0
> Reporter: Adam Binford
> Priority: Minor
>
> Currently, all sorting is disallowed with structured streaming queries. Not allowing global sorting makes sense, but could non-global sorting (i.e. sortWithinPartitions) be allowed? I'm running into this with an external source I'm using, but not sure if this would be useful to file sources as well. I have to foreachBatch so that I can do a sortWithinPartitions.
> Two main questions:
> * Does a local sort cause issues with any exactly-once guarantees streaming queries provides? I can't say I know or understand how these semantics work. Or are there other issues I can't think of this would cause?
> * Is the change as simple as changing the unsupported operations check to only look for global sorts instead of all sorts?
> I have built a version that simply changes the unsupported check to only disallow global sorts and it seems to be working. Anything I'm missing or is it this simple?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org