You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Shixiong Zhu (Jira)" <ji...@apache.org> on 2019/08/20 18:30:00 UTC

[jira] [Resolved] (SPARK-28605) Performance regression in SS's foreach

     [ https://issues.apache.org/jira/browse/SPARK-28605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shixiong Zhu resolved SPARK-28605.
----------------------------------
    Resolution: Invalid

> Performance regression in SS's foreach
> --------------------------------------
>
>                 Key: SPARK-28605
>                 URL: https://issues.apache.org/jira/browse/SPARK-28605
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.4.0, 2.4.1, 2.4.2, 2.4.3
>            Reporter: Shixiong Zhu
>            Priority: Major
>              Labels: regresssion
>
> When "ForeachWriter.open" return "false", ForeachSink v1 will skip the whole partition without reading data. But in ForeachSink v2, due to the API limitation, it needs to read the whole partition even if all data just gets dropped.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org