You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mario Briggs (JIRA)" <ji...@apache.org> on 2016/07/14 10:53:20 UTC
[jira] [Commented] (SPARK-16545) Structured Streaming : foreachSink
creates the Physical Plan multiple times per TriggerInterval
[ https://issues.apache.org/jira/browse/SPARK-16545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376758#comment-15376758 ]
Mario Briggs commented on SPARK-16545:
--------------------------------------
While looking at the performance of Structured streaming, found some excessive time being spent in the driver.
Further looking into this, found the time spent in multiple (3 to be exact) initialisations of QueryExecution.executedPlan due to multiple instances of QueryExecution created in the forEachSink.addBatch.
Creation of physical plan involves more time and hence shouldn't be done more than once
> Structured Streaming : foreachSink creates the Physical Plan multiple times per TriggerInterval
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-16545
> URL: https://issues.apache.org/jira/browse/SPARK-16545
> Project: Spark
> Issue Type: Bug
> Components: SQL, Streaming
> Affects Versions: 2.0.0
> Reporter: Mario Briggs
> Fix For: 2.0.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org