You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Arun Mahadevan (JIRA)" <ji...@apache.org> on 2018/05/25 21:23:00 UTC

[jira] [Commented] (SPARK-23887) update query progress

    [ https://issues.apache.org/jira/browse/SPARK-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491281#comment-16491281 ] 

Arun Mahadevan commented on SPARK-23887:
----------------------------------------

We could probably invoke "ProgressReporter.finishTrigger" after each epoch. This would update the query execution stats and post StreamingQueryListener events. However I am not clear on how the SQL metrics could be updated since it relies on accumulator and the accumulators might not be updated unless the task completes. We could probably post some special event to the DAGScheduler to update the accumulators or need to figure out some other mechanisms.

[~joseph.torres] [~tdas] , what do you think?

> update query progress
> ---------------------
>
>                 Key: SPARK-23887
>                 URL: https://issues.apache.org/jira/browse/SPARK-23887
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>            Reporter: Jose Torres
>            Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org