You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Arun Patel <ar...@gmail.com> on 2015/12/21 04:04:07 UTC
Spark Streaming - Number of RDDs in Dstream
It may be simple question...But, I am struggling to understand this....
DStream is a sequence of RDDs created in a batch window. So, how do I know
how many RDDs are created in a batch?
I am clear about the number of partitions created which is
Number of Partitions = (Batch Interval / spark.streaming.blockInterval) *
number of receivers
Is it like one RDD per receiver? or Multiple RDDs per receiver? What is the
easiest way to find it?
Arun
Re: Spark Streaming - Number of RDDs in Dstream
Posted by Saisai Shao <sa...@gmail.com>.
Yes, basically from the currently implementation it should be.
On Mon, Dec 21, 2015 at 6:39 PM, Arun Patel <ar...@gmail.com> wrote:
> So, Does that mean only one RDD is created by all receivers?
>
>
>
> On Sun, Dec 20, 2015 at 10:23 PM, Saisai Shao <sa...@gmail.com>
> wrote:
>
>> Normally there will be one RDD in each batch.
>>
>> You could refer to the implementation of DStream#getOrCompute.
>>
>>
>> On Mon, Dec 21, 2015 at 11:04 AM, Arun Patel <ar...@gmail.com>
>> wrote:
>>
>>> It may be simple question...But, I am struggling to understand this....
>>>
>>> DStream is a sequence of RDDs created in a batch window. So, how do I
>>> know how many RDDs are created in a batch?
>>>
>>> I am clear about the number of partitions created which is
>>>
>>> Number of Partitions = (Batch Interval / spark.streaming.blockInterval)
>>> * number of receivers
>>>
>>> Is it like one RDD per receiver? or Multiple RDDs per receiver? What is
>>> the easiest way to find it?
>>>
>>> Arun
>>>
>>
>>
>
Re: Spark Streaming - Number of RDDs in Dstream
Posted by Arun Patel <ar...@gmail.com>.
So, Does that mean only one RDD is created by all receivers?
On Sun, Dec 20, 2015 at 10:23 PM, Saisai Shao <sa...@gmail.com>
wrote:
> Normally there will be one RDD in each batch.
>
> You could refer to the implementation of DStream#getOrCompute.
>
>
> On Mon, Dec 21, 2015 at 11:04 AM, Arun Patel <ar...@gmail.com>
> wrote:
>
>> It may be simple question...But, I am struggling to understand this....
>>
>> DStream is a sequence of RDDs created in a batch window. So, how do I
>> know how many RDDs are created in a batch?
>>
>> I am clear about the number of partitions created which is
>>
>> Number of Partitions = (Batch Interval / spark.streaming.blockInterval)
>> * number of receivers
>>
>> Is it like one RDD per receiver? or Multiple RDDs per receiver? What is
>> the easiest way to find it?
>>
>> Arun
>>
>
>
Re: Spark Streaming - Number of RDDs in Dstream
Posted by Saisai Shao <sa...@gmail.com>.
Normally there will be one RDD in each batch.
You could refer to the implementation of DStream#getOrCompute.
On Mon, Dec 21, 2015 at 11:04 AM, Arun Patel <ar...@gmail.com>
wrote:
> It may be simple question...But, I am struggling to understand this....
>
> DStream is a sequence of RDDs created in a batch window. So, how do I
> know how many RDDs are created in a batch?
>
> I am clear about the number of partitions created which is
>
> Number of Partitions = (Batch Interval / spark.streaming.blockInterval) *
> number of receivers
>
> Is it like one RDD per receiver? or Multiple RDDs per receiver? What is
> the easiest way to find it?
>
> Arun
>