You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/07/18 04:05:00 UTC

[jira] [Commented] (SPARK-24386) implement continuous processing coalesce(1)

    [ https://issues.apache.org/jira/browse/SPARK-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547345#comment-16547345 ] 

Apache Spark commented on SPARK-24386:
--------------------------------------

User 'ueshin' has created a pull request for this issue:
https://github.com/apache/spark/pull/21801

> implement continuous processing coalesce(1)
> -------------------------------------------
>
>                 Key: SPARK-24386
>                 URL: https://issues.apache.org/jira/browse/SPARK-24386
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>            Reporter: Jose Torres
>            Assignee: Jose Torres
>            Priority: Major
>             Fix For: 2.4.0
>
>
> [~marmbrus] suggested this as a good implementation checkpoint. If we do the shuffle reader and writer correctly, it should be easy to make a custom coalesce(1) execution for continuous processing using them, without having to implement the logic for shuffle writers finding out where shuffle readers are located. (The coalesce(1) can just get the RpcEndpointRef directly from the reader and pass it to the writers.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org