Reliability of JMS Custom Receiver in Spark Streaming JMS

[Sourav Mazumder <so...@gmail.com>]

Hi,

Need to get bit more understanding of reliability aspects of the Custom
Receivers in the context of the code in spark-streaming-jms
https://github.com/mattf/spark-streaming-jms.

Based on the documentation in
http://spark.apache.org/docs/latest/streaming-custom-receivers.html#receiver-reliability,
I understand that if the store api is called with multiple records the
message is reliably stored as it is a blocking call. On the other hand if
the store api is called with a single record then it is not reliable as the
call is returned back to the calling program before the message is stored
appropriately.

Given that I have few questions

1. Which are the store APIs that relate to multiple records ? Are they the
ones which use scala.collection.mutable.ArrayBuffer<T
<http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/receiver/Receiver.html>>,

scala.collection.Iterator<T
<http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/receiver/Receiver.html>>
and
java.util.Iterator<T
<http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/receiver/Receiver.html>>
in the parameter signature?

2. Is there a sample code which can show how to create multiple records
like that and send the same to appropriate store API ?

3. If I take the example of spark-streaming-jms, the onMessage method of
JMSReceiver class calls store API with one JMSEvent. Does that mean that
this code does not guarantee the reliability of storage of the message
received even if storage level specified to MEMORY_AND_DISK_SER_2 ?

Regards,
Sourav

Re: Reliability of JMS Custom Receiver in Spark Streaming JMS

[Sourav Mazumder <so...@gmail.com>]

Any inputs on this issue ?

Regards,
Sourav

On Tue, May 10, 2016 at 6:17 PM, Sourav Mazumder <
sourav.mazumder00@gmail.com> wrote:

> Hi,
>
> Need to get bit more understanding of reliability aspects of the Custom
> Receivers in the context of the code in spark-streaming-jms
> https://github.com/mattf/spark-streaming-jms.
>
> Based on the documentation in
> http://spark.apache.org/docs/latest/streaming-custom-receivers.html#receiver-reliability,
> I understand that if the store api is called with multiple records the
> message is reliably stored as it is a blocking call. On the other hand if
> the store api is called with a single record then it is not reliable as the
> call is returned back to the calling program before the message is stored
> appropriately.
>
> Given that I have few questions
>
> 1. Which are the store APIs that relate to multiple records ? Are they the
> ones which use scala.collection.mutable.ArrayBuffer<T
> <http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/receiver/Receiver.html>>,
>
> scala.collection.Iterator<T
> <http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/receiver/Receiver.html>>
> and
> java.util.Iterator<T
> <http://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/receiver/Receiver.html>>
> in the parameter signature?
>
> 2. Is there a sample code which can show how to create multiple records
> like that and send the same to appropriate store API ?
>
> 3. If I take the example of spark-streaming-jms, the onMessage method of
> JMSReceiver class calls store API with one JMSEvent. Does that mean that
> this code does not guarantee the reliability of storage of the message
> received even if storage level specified to MEMORY_AND_DISK_SER_2 ?
>
> Regards,
> Sourav
>