You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Ganelin, Ilya" <Il...@capitalone.com> on 2015/07/08 23:23:42 UTC
Real-time data visualization with Zeppelin
Hi all – I’m just wondering if anyone has had success integrating Spark Streaming with Zeppelin and actually dynamically updating the data in near real-time. From my investigation, it seems that Zeppelin will only allow you to display a snapshot of data, not a continuously updating table. Has anyone figured out if there’s a way to loop a display command or how to provide a mechanism to continuously update visualizations?
Thank you,
Ilya Ganelin
[cid:0042A8D7-6242-41E8-80ED-0D0CC16C96B5]
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Re: Real-time data visualization with Zeppelin
Posted by andy petrella <an...@gmail.com>.
Yep, most of the things will work just by renaming it :-D
You can even use nbconvert afterwards
On Thu, Aug 6, 2015 at 12:09 PM jun <ki...@126.com> wrote:
> Hi andy,
>
> Is there any method to convert ipython notebook file(.ipynb) to spark
> notebook file(.snb) or vice versa?
>
> BR
> Jun
>
> At 2015-07-13 02:45:57, "andy petrella" <an...@gmail.com> wrote:
>
> Heya,
>
> You might be looking for something like this I guess:
> https://www.youtube.com/watch?v=kB4kRQRFAVc.
>
> The Spark-Notebook (https://github.com/andypetrella/spark-notebook/) can
> bring that to you actually, it uses fully reactive bilateral communication
> streams to update data and viz, plus it hides almost everything for you ^^.
> The video was using the notebook notebooks/streaming/Twitter stream.snb
> <https://github.com/andypetrella/spark-notebook/blob/master/notebooks/streaming/Twitter%20stream.snb> so
> you can play it yourself if you like.
>
> You might want building the master (before 0.6.0 will be released → soon)
> here http://spark-notebook.io/.
>
> HTH
> andy
>
>
>
> On Sun, Jul 12, 2015 at 8:29 PM Ruslan Dautkhanov <da...@gmail.com>
> wrote:
>
>> Don't think it is a Zeppelin problem.. RDDs are "immutable".
>> Unless you integrate something like IndexedRDD
>> http://spark-packages.org/package/amplab/spark-indexedrdd
>> into Zeppelin I think it's not possible.
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>> On Wed, Jul 8, 2015 at 3:24 PM, Brandon White <bw...@gmail.com>
>> wrote:
>>
>>> Can you use a con job to update it every X minutes?
>>>
>>> On Wed, Jul 8, 2015 at 2:23 PM, Ganelin, Ilya <
>>> Ilya.Ganelin@capitalone.com> wrote:
>>>
>>>> Hi all – I’m just wondering if anyone has had success integrating Spark
>>>> Streaming with Zeppelin and actually dynamically updating the data in near
>>>> real-time. From my investigation, it seems that Zeppelin will only allow
>>>> you to display a snapshot of data, not a continuously updating table. Has
>>>> anyone figured out if there’s a way to loop a display command or how to
>>>> provide a mechanism to continuously update visualizations?
>>>>
>>>> Thank you,
>>>> Ilya Ganelin
>>>>
>>>> [image: 2DD951D6-FF99-4415-80AA-E30EFE7CF452[4].png]
>>>>
>>>> ------------------------------
>>>>
>>>> The information contained in this e-mail is confidential and/or
>>>> proprietary to Capital One and/or its affiliates and may only be used
>>>> solely in performance of work or services for Capital One. The information
>>>> transmitted herewith is intended only for use by the individual or entity
>>>> to which it is addressed. If the reader of this message is not the intended
>>>> recipient, you are hereby notified that any review, retransmission,
>>>> dissemination, distribution, copying or other use of, or taking of any
>>>> action in reliance upon this information is strictly prohibited. If you
>>>> have received this communication in error, please contact the sender and
>>>> delete the material from your computer.
>>>>
>>>
>>>
>> --
andy
Re:Re: Real-time data visualization with Zeppelin
Posted by jun <ki...@126.com>.
Hi andy,
Is there any method to convert ipython notebook file(.ipynb) to spark notebook file(.snb) or vice versa?
BR
Jun
At 2015-07-13 02:45:57, "andy petrella" <an...@gmail.com> wrote:
Heya,
You might be looking for something like this I guess: https://www.youtube.com/watch?v=kB4kRQRFAVc.
The Spark-Notebook (https://github.com/andypetrella/spark-notebook/) can bring that to you actually, it uses fully reactive bilateral communication streams to update data and viz, plus it hides almost everything for you ^^. The video was using the notebook notebooks/streaming/Twitter stream.snb so you can play it yourself if you like.
You might want building the master (before 0.6.0 will be released → soon) here http://spark-notebook.io/.
HTH
andy
On Sun, Jul 12, 2015 at 8:29 PM Ruslan Dautkhanov <da...@gmail.com> wrote:
Don't think it is a Zeppelin problem.. RDDs are "immutable".
Unless you integrate something like IndexedRDD http://spark-packages.org/package/amplab/spark-indexedrdd
into Zeppelin I think it's not possible.
--
Ruslan Dautkhanov
On Wed, Jul 8, 2015 at 3:24 PM, Brandon White <bw...@gmail.com> wrote:
Can you use a con job to update it every X minutes?
On Wed, Jul 8, 2015 at 2:23 PM, Ganelin, Ilya <Il...@capitalone.com> wrote:
Hi all – I’m just wondering if anyone has had success integrating Spark Streaming with Zeppelin and actually dynamically updating the data in near real-time. From my investigation, it seems that Zeppelin will only allow you to display a snapshot of data, not a continuously updating table. Has anyone figured out if there’s a way to loop a display command or how to provide a mechanism to continuously update visualizations?
Thank you,
Ilya Ganelin
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Real-time data visualization with Zeppelin
Posted by andy petrella <an...@gmail.com>.
Heya,
You might be looking for something like this I guess:
https://www.youtube.com/watch?v=kB4kRQRFAVc.
The Spark-Notebook (https://github.com/andypetrella/spark-notebook/) can
bring that to you actually, it uses fully reactive bilateral communication
streams to update data and viz, plus it hides almost everything for you ^^.
The video was using the notebook notebooks/streaming/Twitter stream.snb
<https://github.com/andypetrella/spark-notebook/blob/master/notebooks/streaming/Twitter%20stream.snb>
so
you can play it yourself if you like.
You might want building the master (before 0.6.0 will be released → soon)
here http://spark-notebook.io/.
HTH
andy
On Sun, Jul 12, 2015 at 8:29 PM Ruslan Dautkhanov <da...@gmail.com>
wrote:
> Don't think it is a Zeppelin problem.. RDDs are "immutable".
> Unless you integrate something like IndexedRDD
> http://spark-packages.org/package/amplab/spark-indexedrdd
> into Zeppelin I think it's not possible.
>
>
> --
> Ruslan Dautkhanov
>
> On Wed, Jul 8, 2015 at 3:24 PM, Brandon White <bw...@gmail.com>
> wrote:
>
>> Can you use a con job to update it every X minutes?
>>
>> On Wed, Jul 8, 2015 at 2:23 PM, Ganelin, Ilya <
>> Ilya.Ganelin@capitalone.com> wrote:
>>
>>> Hi all – I’m just wondering if anyone has had success integrating Spark
>>> Streaming with Zeppelin and actually dynamically updating the data in near
>>> real-time. From my investigation, it seems that Zeppelin will only allow
>>> you to display a snapshot of data, not a continuously updating table. Has
>>> anyone figured out if there’s a way to loop a display command or how to
>>> provide a mechanism to continuously update visualizations?
>>>
>>> Thank you,
>>> Ilya Ganelin
>>>
>>> [image: 2DD951D6-FF99-4415-80AA-E30EFE7CF452[4].png]
>>>
>>> ------------------------------
>>>
>>> The information contained in this e-mail is confidential and/or
>>> proprietary to Capital One and/or its affiliates and may only be used
>>> solely in performance of work or services for Capital One. The information
>>> transmitted herewith is intended only for use by the individual or entity
>>> to which it is addressed. If the reader of this message is not the intended
>>> recipient, you are hereby notified that any review, retransmission,
>>> dissemination, distribution, copying or other use of, or taking of any
>>> action in reliance upon this information is strictly prohibited. If you
>>> have received this communication in error, please contact the sender and
>>> delete the material from your computer.
>>>
>>
>>
>
Re: Real-time data visualization with Zeppelin
Posted by Ruslan Dautkhanov <da...@gmail.com>.
Don't think it is a Zeppelin problem.. RDDs are "immutable".
Unless you integrate something like IndexedRDD
http://spark-packages.org/package/amplab/spark-indexedrdd
into Zeppelin I think it's not possible.
--
Ruslan Dautkhanov
On Wed, Jul 8, 2015 at 3:24 PM, Brandon White <bw...@gmail.com>
wrote:
> Can you use a con job to update it every X minutes?
>
> On Wed, Jul 8, 2015 at 2:23 PM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com
> > wrote:
>
>> Hi all – I’m just wondering if anyone has had success integrating Spark
>> Streaming with Zeppelin and actually dynamically updating the data in near
>> real-time. From my investigation, it seems that Zeppelin will only allow
>> you to display a snapshot of data, not a continuously updating table. Has
>> anyone figured out if there’s a way to loop a display command or how to
>> provide a mechanism to continuously update visualizations?
>>
>> Thank you,
>> Ilya Ganelin
>>
>>
>> ------------------------------
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>
>
Re: Real-time data visualization with Zeppelin
Posted by Brandon White <bw...@gmail.com>.
Can you use a con job to update it every X minutes?
On Wed, Jul 8, 2015 at 2:23 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Hi all – I’m just wondering if anyone has had success integrating Spark
> Streaming with Zeppelin and actually dynamically updating the data in near
> real-time. From my investigation, it seems that Zeppelin will only allow
> you to display a snapshot of data, not a continuously updating table. Has
> anyone figured out if there’s a way to loop a display command or how to
> provide a mechanism to continuously update visualizations?
>
> Thank you,
> Ilya Ganelin
>
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>