More about the use case

[Hao Ren <in...@gmail.com>]

Hi, I am new to zeppelin, come from spark background.

I would like to know whether zeppelin supports the following use case:


Given a spark application (*not written in zeppelin*, *just a jar, etc*.),
I would like to sent a DateFrame to a zeppelin server while the spark job
is running, and the zeppelin displays the data in any kind of chart once it
receives data from spark.

In this case, zeppelin is just like a chart server which waits for some
inputs from a spark job apart. The inputs can be a DataFrame, scala
collection, etc.

If any implicit APIs implemented for spark RDD or DataFrame exists, it
would be like the following:

// In my spark job
dataFrame.sendToZeppelin(zplMasterURL)


I need this feature because we have created a spark application for machine
learning, and want to visualize the result. It is not possible to copy all
code in zeppelin note book. We don't want load our application jar in
zeppelin, neither. We prefer some zeppelin APIs allows us to send data to a
zeppelin server from our spark job apart.

Zeppelin chart is very clear and reactive, we really want to take advantage
of this and avoid reinventing the wheel.

Any help is highly appreciated.

Thank you.

-- 
Hao Ren

Data Engineer @ leboncoin

Paris, France

Re: More about the use case

[moon soo Lee <mo...@apache.org>]

Hi Hao Ren,

Thanks for sharing your use case.

The simplest way i know is persist DataFrame from your spark application
and then read the data from Zeppelin. OFF_HEAP with Tachyon may help.

Best,
moon




On Tue, Sep 22, 2015 at 6:36 PM Hao Ren <in...@gmail.com> wrote:

> Hi, I am new to zeppelin, come from spark background.
>
> I would like to know whether zeppelin supports the following use case:
>
>
> Given a spark application (*not written in zeppelin*, *just a jar, etc*.),
> I would like to sent a DateFrame to a zeppelin server while the spark job
> is running, and the zeppelin displays the data in any kind of chart once it
> receives data from spark.
>
> In this case, zeppelin is just like a chart server which waits for some
> inputs from a spark job apart. The inputs can be a DataFrame, scala
> collection, etc.
>
> If any implicit APIs implemented for spark RDD or DataFrame exists, it
> would be like the following:
>
> // In my spark job
> dataFrame.sendToZeppelin(zplMasterURL)
>
>
> I need this feature because we have created a spark application for
> machine learning, and want to visualize the result. It is not possible to
> copy all code in zeppelin note book. We don't want load our application jar
> in zeppelin, neither. We prefer some zeppelin APIs allows us to send data
> to a zeppelin server from our spark job apart.
>
> Zeppelin chart is very clear and reactive, we really want to take
> advantage of this and avoid reinventing the wheel.
>
> Any help is highly appreciated.
>
> Thank you.
>
> --
> Hao Ren
>
> Data Engineer @ leboncoin
>
> Paris, France
>