You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Natalie Ruiz <Na...@microsoft.com.INVALID> on 2019/09/03 22:25:47 UTC
Structured Streaming: How to add a listener for when a batch is
complete
Hello all,
I'm a beginner, new to Spark and wanted to know if there was an equivalent to Spark Streaming's StreamingListenerBatchCompleted in Structured Streaming? I want to add a listener for when a batch is complete but the documentation and examples I find are for Spark Streaming and not Structured Streaming and registering it to a StreamingContext, which from my understanding is just for Spark Streaming
Thanks!
Re: Structured Streaming: How to add a listener for when a batch is complete
Posted by Gourav Sengupta <go...@gmail.com>.
Hi Natalie,
in case we are using Python then listener is not available, we can use the
SPARK UI REST API's .
For that to happen we have to start the SPARK session with following
settings: spark.sql("SET spark.sql.streaming.metricsEnabled=true")
For the UI from which we can use the JSON results from the below:
https://spark.apache.org/docs/latest/monitoring.html#rest-api
In case you are working on AWS EMR, the UI URL is a bit different, let me
know in that case and I will email to you.
Hi TD,
What will be the best way to stop a long running SPARK streaming job
gracefully? I will be really grateful if you could kindly help us out.
Regards,
Gourav Sengupta
On Wed, Sep 4, 2019 at 2:28 AM Tathagata Das <td...@databricks.com> wrote:
>
> https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#reporting-metrics-programmatically-using-asynchronous-apis
>
> On Tue, Sep 3, 2019, 3:26 PM Natalie Ruiz
> <Na...@microsoft.com.invalid> wrote:
>
>> Hello all,
>>
>>
>>
>> I’m a beginner, new to Spark and wanted to know if there was an
>> equivalent to Spark Streaming’s StreamingListenerBatchCompleted in
>> Structured Streaming? I want to add a listener for when a batch is complete
>> but the documentation and examples I find are for Spark Streaming and not
>> Structured Streaming and registering it to a StreamingContext, which from
>> my understanding is just for Spark Streaming
>>
>>
>>
>> Thanks!
>>
>
Re: Structured Streaming: How to add a listener for when a batch is complete
Posted by Tathagata Das <td...@databricks.com>.
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#reporting-metrics-programmatically-using-asynchronous-apis
On Tue, Sep 3, 2019, 3:26 PM Natalie Ruiz
<Na...@microsoft.com.invalid> wrote:
> Hello all,
>
>
>
> I’m a beginner, new to Spark and wanted to know if there was an equivalent
> to Spark Streaming’s StreamingListenerBatchCompleted in Structured
> Streaming? I want to add a listener for when a batch is complete but the
> documentation and examples I find are for Spark Streaming and not
> Structured Streaming and registering it to a StreamingContext, which from
> my understanding is just for Spark Streaming
>
>
>
> Thanks!
>