CQs on WindowedStream created on running StreamingContext

[Yogs <ma...@gmail.com>]

Hi,

We intend to run adhoc windowed continuous queries on spark streaming data.
The queries could be registered/deregistered dynamically or can be
submitted through command line. Currently Spark streaming doesn’t allow
adding any new inputs, transformations, and output operations after
starting a StreamingContext. But doing following code changes in
DStream.scala allows me to create an window on DStream even after
StreamingContext has started (in StreamingContextState.ACTIVE).

1) In DStream.validateAtInit()
Allowed adding new inputs, transformations, and output operations after
starting a streaming context
2) In DStream.persist()
Allowed to change storage level of an DStream after streaming context has
started

Ultimately the window api just does slice on the parentRDD and returns
allRDDsInWindow.
We create DataFrames out of these RDDs from this particular
WindowedDStream, and evaluate queries on those DataFrames.

1) Do you see any challenges and consequences with this approach ?
2) Will these on the fly created WindowedDStreams be accounted properly in
Runtime and memory management?
3) What is the reason we do not allow creating new windows with
StreamingContextState.ACTIVE state?
4) Does it make sense to add our own implementation of WindowedDStream in
this case?

- Yogesh

Re: CQs on WindowedStream created on running StreamingContext

[Yogesh Mahajan <ma...@gmail.com>]

Anyone knows about this ? TD ?

-yogesh

> On 30-Sep-2015, at 1:25 pm, Yogs <ma...@gmail.com> wrote:
> 
> Hi, 
> 
> We intend to run adhoc windowed continuous queries on spark streaming data. The queries could be registered/deregistered dynamically or can be submitted through command line. Currently Spark streaming doesn’t allow adding any new inputs, transformations, and output operations after starting a StreamingContext. But doing following code changes in DStream.scala allows me to create an window on DStream even after StreamingContext has started (in StreamingContextState.ACTIVE). 
> 
> 1) In DStream.validateAtInit()
> Allowed adding new inputs, transformations, and output operations after starting a streaming context
> 2) In DStream.persist()
> Allowed to change storage level of an DStream after streaming context has started
> 
> Ultimately the window api just does slice on the parentRDD and returns allRDDsInWindow.
> We create DataFrames out of these RDDs from this particular WindowedDStream, and evaluate queries on those DataFrames. 
> 
> 1) Do you see any challenges and consequences with this approach ? 
> 2) Will these on the fly created WindowedDStreams be accounted properly in Runtime and memory management?
> 3) What is the reason we do not allow creating new windows with StreamingContextState.ACTIVE state?
> 4) Does it make sense to add our own implementation of WindowedDStream in this case?
> 
> - Yogesh 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org