Re: spark-streaming

[Nathan Kronenfeld <nk...@oculusinfo.com>]

We were using graph.zeroTime, to figure out which files were relevant to
the DStream.

It seems difficult to us to see how one would make a custom DStream without
access to the graph in general though.

And more egregious, the disparity between the privacy and documentation of
clearMetadata and addMetadata was particularly discouraging.



On Wed, Mar 19, 2014 at 7:09 PM, Tathagata Das
<ta...@gmail.com>wrote:

> Hey Nathan,
>
> We made that private in order to reduce the visible public API, to have
> greater control in the future. Can you tell me more about the timing
> information that you want to get?
>
> TD
>
>
> On Fri, Mar 14, 2014 at 8:57 PM, Nathan Kronenfeld <
> nkronenfeld@oculusinfo.com> wrote:
>
>> I'm trying to update some spark streaming code from 0.8.1 to 0.9.0.
>>
>> Among other things, I've found the function clearMetadata, who's comment
>> says:
>>
>>     "...Subclasses of DStream may override this to clear their own
>> metadata along with the generated RDDs"
>>
>> yet which is declared private[streaming].
>>
>> How are subclasses expected to override this if it's private? If they
>> aren't, how and when should they now clear any extraneous data they have?
>>
>> Similarly, I now see no way to get the timing information - how is a
>> custom dstream supposed to do this now?
>>
>> Thanks,
>>                 -Nathan
>>
>>
>>
>> --
>> Nathan Kronenfeld
>> Senior Visualization Developer
>> Oculus Info Inc
>> 2 Berkeley Street, Suite 600,
>> Toronto, Ontario M5A 4J5
>> Phone:  +1-416-203-3003 x 238
>> Email:  nkronenfeld@oculusinfo.com
>>
>
>


-- 
Nathan Kronenfeld
Senior Visualization Developer
Oculus Info Inc
2 Berkeley Street, Suite 600,
Toronto, Ontario M5A 4J5
Phone:  +1-416-203-3003 x 238
Email:  nkronenfeld@oculusinfo.com