You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by avilevi3 <av...@gmail.com> on 2015/02/23 09:57:14 UTC
spark streaming window operations on a large window size
Hi guys,
does spark streaming supports window operations on a sliding window that is
data is larger than the available memory?
we would like to
currently we are using kafka as input, but we could change that if needed.
thanks
Avi
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-window-operations-on-a-large-window-size-tp21764.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark streaming window operations on a large window size
Posted by Avi Levi <av...@gmail.com>.
OK - thanks a lot
On Tue, Feb 24, 2015 at 9:49 AM, Tathagata Das <td...@databricks.com> wrote:
> Yes.
>
> On Mon, Feb 23, 2015 at 11:16 PM, Avi Levi <av...@gmail.com> wrote:
>
>> @Tathagata Das so basically you are saying it is supported out of the
>> box, but we should expect a significant performance hit - is that right?
>>
>>
>>
>> On Tue, Feb 24, 2015 at 5:37 AM, Tathagata Das <td...@databricks.com>
>> wrote:
>>
>>> The default persistence level is MEMORY_AND_DISK, so the LRU policy
>>> would discard the blocks to disk, so the streaming app will not fail.
>>> However, since things will get constantly read in and out of disk as
>>> windows are processed, the performance wont be great. So it is best to have
>>> sufficient memory to keep all the window data in memory.
>>>
>>> TD
>>>
>>> On Mon, Feb 23, 2015 at 8:26 AM, Shao, Saisai <sa...@intel.com>
>>> wrote:
>>>
>>>> I don't think current Spark Streaming supports window operations which
>>>> beyond its available memory, internally Spark Streaming puts all the data
>>>> in the memory belongs to the effective window, if the memory is not enough,
>>>> BlockManager will discard the blocks at LRU policy, so something unexpected
>>>> will be occurred.
>>>>
>>>> Thanks
>>>> Jerry
>>>>
>>>> -----Original Message-----
>>>> From: avilevi3 [mailto:avilevi3@gmail.com]
>>>> Sent: Monday, February 23, 2015 12:57 AM
>>>> To: user@spark.apache.org
>>>> Subject: spark streaming window operations on a large window size
>>>>
>>>> Hi guys,
>>>>
>>>> does spark streaming supports window operations on a sliding window
>>>> that is data is larger than the available memory?
>>>> we would like to
>>>> currently we are using kafka as input, but we could change that if
>>>> needed.
>>>>
>>>> thanks
>>>> Avi
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-window-operations-on-a-large-window-size-tp21764.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For
>>>> additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>
>
Re: spark streaming window operations on a large window size
Posted by Tathagata Das <td...@databricks.com>.
Yes.
On Mon, Feb 23, 2015 at 11:16 PM, Avi Levi <av...@gmail.com> wrote:
> @Tathagata Das so basically you are saying it is supported out of the
> box, but we should expect a significant performance hit - is that right?
>
>
>
> On Tue, Feb 24, 2015 at 5:37 AM, Tathagata Das <td...@databricks.com>
> wrote:
>
>> The default persistence level is MEMORY_AND_DISK, so the LRU policy would
>> discard the blocks to disk, so the streaming app will not fail. However,
>> since things will get constantly read in and out of disk as windows are
>> processed, the performance wont be great. So it is best to have sufficient
>> memory to keep all the window data in memory.
>>
>> TD
>>
>> On Mon, Feb 23, 2015 at 8:26 AM, Shao, Saisai <sa...@intel.com>
>> wrote:
>>
>>> I don't think current Spark Streaming supports window operations which
>>> beyond its available memory, internally Spark Streaming puts all the data
>>> in the memory belongs to the effective window, if the memory is not enough,
>>> BlockManager will discard the blocks at LRU policy, so something unexpected
>>> will be occurred.
>>>
>>> Thanks
>>> Jerry
>>>
>>> -----Original Message-----
>>> From: avilevi3 [mailto:avilevi3@gmail.com]
>>> Sent: Monday, February 23, 2015 12:57 AM
>>> To: user@spark.apache.org
>>> Subject: spark streaming window operations on a large window size
>>>
>>> Hi guys,
>>>
>>> does spark streaming supports window operations on a sliding window that
>>> is data is larger than the available memory?
>>> we would like to
>>> currently we are using kafka as input, but we could change that if
>>> needed.
>>>
>>> thanks
>>> Avi
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-window-operations-on-a-large-window-size-tp21764.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For
>>> additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
Re: spark streaming window operations on a large window size
Posted by Avi Levi <av...@gmail.com>.
@Tathagata Das so basically you are saying it is supported out of the box,
but we should expect a significant performance hit - is that right?
On Tue, Feb 24, 2015 at 5:37 AM, Tathagata Das <td...@databricks.com> wrote:
> The default persistence level is MEMORY_AND_DISK, so the LRU policy would
> discard the blocks to disk, so the streaming app will not fail. However,
> since things will get constantly read in and out of disk as windows are
> processed, the performance wont be great. So it is best to have sufficient
> memory to keep all the window data in memory.
>
> TD
>
> On Mon, Feb 23, 2015 at 8:26 AM, Shao, Saisai <sa...@intel.com>
> wrote:
>
>> I don't think current Spark Streaming supports window operations which
>> beyond its available memory, internally Spark Streaming puts all the data
>> in the memory belongs to the effective window, if the memory is not enough,
>> BlockManager will discard the blocks at LRU policy, so something unexpected
>> will be occurred.
>>
>> Thanks
>> Jerry
>>
>> -----Original Message-----
>> From: avilevi3 [mailto:avilevi3@gmail.com]
>> Sent: Monday, February 23, 2015 12:57 AM
>> To: user@spark.apache.org
>> Subject: spark streaming window operations on a large window size
>>
>> Hi guys,
>>
>> does spark streaming supports window operations on a sliding window that
>> is data is larger than the available memory?
>> we would like to
>> currently we are using kafka as input, but we could change that if needed.
>>
>> thanks
>> Avi
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-window-operations-on-a-large-window-size-tp21764.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
>> commands, e-mail: user-help@spark.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: spark streaming window operations on a large window size
Posted by Tathagata Das <td...@databricks.com>.
The default persistence level is MEMORY_AND_DISK, so the LRU policy would
discard the blocks to disk, so the streaming app will not fail. However,
since things will get constantly read in and out of disk as windows are
processed, the performance wont be great. So it is best to have sufficient
memory to keep all the window data in memory.
TD
On Mon, Feb 23, 2015 at 8:26 AM, Shao, Saisai <sa...@intel.com> wrote:
> I don't think current Spark Streaming supports window operations which
> beyond its available memory, internally Spark Streaming puts all the data
> in the memory belongs to the effective window, if the memory is not enough,
> BlockManager will discard the blocks at LRU policy, so something unexpected
> will be occurred.
>
> Thanks
> Jerry
>
> -----Original Message-----
> From: avilevi3 [mailto:avilevi3@gmail.com]
> Sent: Monday, February 23, 2015 12:57 AM
> To: user@spark.apache.org
> Subject: spark streaming window operations on a large window size
>
> Hi guys,
>
> does spark streaming supports window operations on a sliding window that
> is data is larger than the available memory?
> we would like to
> currently we are using kafka as input, but we could change that if needed.
>
> thanks
> Avi
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-window-operations-on-a-large-window-size-tp21764.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional
> commands, e-mail: user-help@spark.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
RE: spark streaming window operations on a large window size
Posted by "Shao, Saisai" <sa...@intel.com>.
I don't think current Spark Streaming supports window operations which beyond its available memory, internally Spark Streaming puts all the data in the memory belongs to the effective window, if the memory is not enough, BlockManager will discard the blocks at LRU policy, so something unexpected will be occurred.
Thanks
Jerry
-----Original Message-----
From: avilevi3 [mailto:avilevi3@gmail.com]
Sent: Monday, February 23, 2015 12:57 AM
To: user@spark.apache.org
Subject: spark streaming window operations on a large window size
Hi guys,
does spark streaming supports window operations on a sliding window that is data is larger than the available memory?
we would like to
currently we are using kafka as input, but we could change that if needed.
thanks
Avi
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-window-operations-on-a-large-window-size-tp21764.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For additional commands, e-mail: user-help@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org