You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jose Torres (JIRA)" <ji...@apache.org> on 2018/05/24 23:52:00 UTC

[jira] [Created] (SPARK-24386) implement continuous processing coalesce(1)

Jose Torres created SPARK-24386:
-----------------------------------

             Summary: implement continuous processing coalesce(1)
                 Key: SPARK-24386
                 URL: https://issues.apache.org/jira/browse/SPARK-24386
             Project: Spark
          Issue Type: Sub-task
          Components: Structured Streaming
    Affects Versions: 2.4.0
            Reporter: Jose Torres


[~marmbrus] suggested this as a good implementation checkpoint. If we do the shuffle reader and writer correctly, it should be easy to make a custom coalesce(1) execution for continuous processing using them, without having to implement the logic for shuffle writers finding out where shuffle readers are located. (The coalesce(1) can just get the RpcEndpointRef directly from the reader and pass it to the writers.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org