[jira] [Assigned] (SPARK-24933) SinkProgress should report written rows

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-24933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-24933:
------------------------------------

    Assignee:     (was: Apache Spark)

> SinkProgress should report written rows
> ---------------------------------------
>
>                 Key: SPARK-24933
>                 URL: https://issues.apache.org/jira/browse/SPARK-24933
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.3.1
>            Reporter: Vaclav Kosar
>            Priority: Major
>
> SinkProgress should report similar properties like SourceProgress as long as they are available for given Sink. Count of written rows is metric availble for all Sinks. Since relevant progress information is with respect to commited rows, ideal object to carry this info is WriterCommitMessage. For brevity the implementation will focus only on Sinks with API V2 and on Micro Batch mode. Implemention for Continuous mode will be provided at later date.
> h4. Before
> {code}
> {"description":"org.apache.spark.sql.kafka010.KafkaSourceProvider@3c0bd317"}
> {code}
> h4. After
> {code}
> {"description":"org.apache.spark.sql.kafka010.KafkaSourceProvider@3c0bd317","numOutputRows":5000}
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org