[jira] [Created] (SPARK-31376) Non-global sort support for structured streaming

["Adam Binford (Jira)" <ji...@apache.org>]

Adam Binford created SPARK-31376:
------------------------------------

             Summary: Non-global sort support for structured streaming
                 Key: SPARK-31376
                 URL: https://issues.apache.org/jira/browse/SPARK-31376
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.0.0
            Reporter: Adam Binford


Currently, all sorting is disallowed with structured streaming queries. Not allowing global sorting makes sense, but could non-global sorting (i.e. sortWithinPartitions) be allowed? I'm running into this with an external source I'm using, but not sure if this would be useful to file sources as well. I have to foreachBatch so that I can do a sortWithinPartitions.

Two main questions:
 * Does a local sort cause issues with any exactly-once guarantees streaming queries provides? I can't say I know or understand how these semantics work. Or are there other issues I can't think of this would cause?
 * Is the change as simple as changing the unsupported operations check to only look for global sorts instead of all sorts?

I have built a version that simply changes the unsupported check to only disallow global sorts and it seems to be working. Anything I'm missing or is it this simple?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org