You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tathagata Das (JIRA)" <ji...@apache.org> on 2015/04/30 00:41:06 UTC
[jira] [Comment Edited] (SPARK-7111) Add a tracker to track the
direct (receiver-less) streams
[ https://issues.apache.org/jira/browse/SPARK-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520432#comment-14520432 ]
Tathagata Das edited comment on SPARK-7111 at 4/29/15 10:40 PM:
----------------------------------------------------------------
Here is the design that I propose. If this looks good, then mind implementing this quickly? :) Spark 1.4 code freeze is really close.
https://docs.google.com/document/d/122QvcwPoLkI2OW4eM7nyBOAqffk2uxgsNT38WI-M5vQ/edit#heading=h.9eluy73ulzuz
Also looping in [~zsxwing] as he ultimately uses {{BatchInfo}} to generate the updated streaming UI in https://github.com/apache/spark/pull/5533
was (Author: tdas):
Here is the design that I propose. If this looks good, then mind implementing this quickly? :) Spark 1.4 code freeze is really close.
Also looping in [~zsxwing] as he ultimately uses {{BatchInfo}} to generate the updated streaming UI in https://github.com/apache/spark/pull/5533
> Add a tracker to track the direct (receiver-less) streams
> ---------------------------------------------------------
>
> Key: SPARK-7111
> URL: https://issues.apache.org/jira/browse/SPARK-7111
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: Saisai Shao
>
> Currently for receiver-based input streams, Spark Streaming offers ReceiverTracker and ReceivedBlockTracker to track the status of receivers as well as block information. Also this status and block information can be retrieved from StreamingListener to expose to the users.
> But for direct-based (receiver-less) input streams, Current Spark Streaming lacks such mechanism to track the registered direct streams, also lacks the way to track the processed number of data for direct-based input streams.
> Here propose a mechanism to track the register direct stream, also expose the processing statistics to the BatchInfo and StreamingListener.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org