You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Cameron McBride <ca...@gmail.com> on 2016/06/16 15:54:16 UTC
zeppelin PostgreSQL interpreter data sharing
Greetings,
I'm brand new to zeppelin, and this notebook technology looks great. I'm
evaluating using it for our data science team, but got it up and running
quickly using some PostgreSQL data and some spark tests. The distributed
nature of each paragraph, and naturally varying interpreters within a
notebook is fantastic. Some of the current built in technology is so
refreshing compared with trying to beat in such functionality to other
notebook / sharing solutions. Anyhow, I'm stoked and wanted to share that
first as part of my first post. :)
In using the PostgreSQL interpreter, is there a way to share some output
data with other paragraphs? I'm trying to run a SQL query, then perhaps do
some analysis or derived logic in a different interpreter later in the
notebook. Sorry if I missed this somewhere, but my digging couldn't find a
workable answer.
TIA,
Cameron
Re: zeppelin PostgreSQL interpreter data sharing
Posted by moon soo Lee <mo...@apache.org>.
There is really basic usage[1] in Spark interpreter docs. [2] is
corresponding source code.
And this diagram [3] from Helium proposal [4] might help little bit more.
Currently, DistributedResourcePool [5] implements ResourcePool api.
DistributedResourcePool keeps the data in the individual interpreter
process.
When other interpreter process trying to access the data, the data is being
serialized and transferred to other interpreter process and deserialized.
Data can be arbitrary java object. If object is not serializable, then
it'll be accessible from the same interpreter process only.
AngularObject is persisted into notebook json file. But objects in the
ResourcePool is volatile for now.
Please feel free to ask any further questions.
Thanks,
moon
[1]
http://zeppelin.incubator.apache.org/docs/0.6.0-SNAPSHOT/interpreter/spark.html#object-exchange
[2]
https://github.com/apache/zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/ZeppelinContext.java#L694
[3]
https://cwiki.apache.org/confluence/download/attachments/61318320/image2015-9-9%2010%3A36%3A28.png?version=1&modificationDate=1441820395000&api=v2
[4] https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal
[5]
https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/resource/DistributedResourcePool.java
On Thu, Jun 16, 2016 at 9:53 AM Cameron McBride <ca...@gmail.com>
wrote:
> Appreciate the quick responses!
>
> The patch (#pr836) does sound like a good solution, as it will generically
> address this issue (and makes use of the existing ResourcePool framework).
> The example of this working with the shell is also nice. Obviously my vote
> is to include it in 0.6.0. :)
>
> Are there any good examples on how to use the resource pool? Or a pointer
> to some documentation / guide / notes of what data lives where (within
> interpreter, Angular, ResourcePool, etc.)?
>
> Cameron
>
>
>
> On Thu, Jun 16, 2016 at 12:30 PM, moon soo Lee <mo...@apache.org> wrote:
>
>> ResourcePool [1] is a data store that all the different types of
>> interpreter can access.
>> It allows exchange data between paragraphs/notebooks/interpreters.
>>
>> However, not all interpreter uses ResourcePool, at the moment.
>> For example Spark interpreter family provides API to use it (spark,
>> pyspark, sparkr), which is z.put(), z.get(), but all other interpreter
>> implementation does not leverage ResourcePool at the moment.
>>
>> We can simply modify PostgreSQL interpreter to push result into
>> ResourcePool, so other interpreters, for example SparkInterpreter, can get
>> result from ResourcePool and process. I think this implementation will
>> be as simple as few lines of code.
>>
>> Alternative approach is #pr836[2] (hope we can merge this, soon). This
>> patch includes the code that automatically pushes all the results into
>> ResourcePool, without modification of each interpreter.
>>
>> Thanks,
>> moon
>>
>> [1]
>> https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/resource/ResourcePool.java
>> [2] https://github.com/apache/zeppelin/pull/836
>>
>>
>> On Thu, Jun 16, 2016 at 9:13 AM Mohit Jaggi <mo...@gmail.com> wrote:
>>
>>> I believe z.context() is the only way to share data between
>>> interpreters. Within an interpreter data is usually available across
>>> paragraphs…perhaps even across notebooks as I guess zeppelin will create a
>>> single interpreter in the backend unless you somehow make it use separate
>>> ones.
>>>
>>> > On Jun 16, 2016, at 9:24 PM, Cameron McBride <
>>> cameron.mcbride@gmail.com> wrote:
>>> >
>>> > Greetings,
>>> >
>>> > I'm brand new to zeppelin, and this notebook technology looks great.
>>> I'm evaluating using it for our data science team, but got it up and
>>> running quickly using some PostgreSQL data and some spark tests. The
>>> distributed nature of each paragraph, and naturally varying interpreters
>>> within a notebook is fantastic. Some of the current built in technology is
>>> so refreshing compared with trying to beat in such functionality to other
>>> notebook / sharing solutions. Anyhow, I'm stoked and wanted to share that
>>> first as part of my first post. :)
>>> >
>>> > In using the PostgreSQL interpreter, is there a way to share some
>>> output data with other paragraphs? I'm trying to run a SQL query, then
>>> perhaps do some analysis or derived logic in a different interpreter later
>>> in the notebook. Sorry if I missed this somewhere, but my digging couldn't
>>> find a workable answer.
>>> >
>>> > TIA,
>>> >
>>> > Cameron
>>> >
>>> >
>>>
>>>
>
Re: zeppelin PostgreSQL interpreter data sharing
Posted by Mohit Jaggi <mo...@gmail.com>.
Moon: Are z. Context and resource pool different or the same?
Sent from my iPhone
> On Jun 16, 2016, at 10:23 PM, Cameron McBride <ca...@gmail.com> wrote:
>
> Appreciate the quick responses!
>
> The patch (#pr836) does sound like a good solution, as it will generically address this issue (and makes use of the existing ResourcePool framework). The example of this working with the shell is also nice. Obviously my vote is to include it in 0.6.0. :)
>
> Are there any good examples on how to use the resource pool? Or a pointer to some documentation / guide / notes of what data lives where (within interpreter, Angular, ResourcePool, etc.)?
>
> Cameron
>
>
>
>> On Thu, Jun 16, 2016 at 12:30 PM, moon soo Lee <mo...@apache.org> wrote:
>> ResourcePool [1] is a data store that all the different types of interpreter can access.
>> It allows exchange data between paragraphs/notebooks/interpreters.
>>
>> However, not all interpreter uses ResourcePool, at the moment.
>> For example Spark interpreter family provides API to use it (spark, pyspark, sparkr), which is z.put(), z.get(), but all other interpreter implementation does not leverage ResourcePool at the moment.
>>
>> We can simply modify PostgreSQL interpreter to push result into ResourcePool, so other interpreters, for example SparkInterpreter, can get result from ResourcePool and process. I think this implementation will be as simple as few lines of code.
>>
>> Alternative approach is #pr836[2] (hope we can merge this, soon). This patch includes the code that automatically pushes all the results into ResourcePool, without modification of each interpreter.
>>
>> Thanks,
>> moon
>>
>> [1] https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/resource/ResourcePool.java
>> [2] https://github.com/apache/zeppelin/pull/836
>>
>>
>>> On Thu, Jun 16, 2016 at 9:13 AM Mohit Jaggi <mo...@gmail.com> wrote:
>>> I believe z.context() is the only way to share data between interpreters. Within an interpreter data is usually available across paragraphs…perhaps even across notebooks as I guess zeppelin will create a single interpreter in the backend unless you somehow make it use separate ones.
>>>
>>> > On Jun 16, 2016, at 9:24 PM, Cameron McBride <ca...@gmail.com> wrote:
>>> >
>>> > Greetings,
>>> >
>>> > I'm brand new to zeppelin, and this notebook technology looks great. I'm evaluating using it for our data science team, but got it up and running quickly using some PostgreSQL data and some spark tests. The distributed nature of each paragraph, and naturally varying interpreters within a notebook is fantastic. Some of the current built in technology is so refreshing compared with trying to beat in such functionality to other notebook / sharing solutions. Anyhow, I'm stoked and wanted to share that first as part of my first post. :)
>>> >
>>> > In using the PostgreSQL interpreter, is there a way to share some output data with other paragraphs? I'm trying to run a SQL query, then perhaps do some analysis or derived logic in a different interpreter later in the notebook. Sorry if I missed this somewhere, but my digging couldn't find a workable answer.
>>> >
>>> > TIA,
>>> >
>>> > Cameron
>>> >
>>> >
>
Re: zeppelin PostgreSQL interpreter data sharing
Posted by Cameron McBride <ca...@gmail.com>.
Appreciate the quick responses!
The patch (#pr836) does sound like a good solution, as it will generically
address this issue (and makes use of the existing ResourcePool framework).
The example of this working with the shell is also nice. Obviously my vote
is to include it in 0.6.0. :)
Are there any good examples on how to use the resource pool? Or a pointer
to some documentation / guide / notes of what data lives where (within
interpreter, Angular, ResourcePool, etc.)?
Cameron
On Thu, Jun 16, 2016 at 12:30 PM, moon soo Lee <mo...@apache.org> wrote:
> ResourcePool [1] is a data store that all the different types of
> interpreter can access.
> It allows exchange data between paragraphs/notebooks/interpreters.
>
> However, not all interpreter uses ResourcePool, at the moment.
> For example Spark interpreter family provides API to use it (spark,
> pyspark, sparkr), which is z.put(), z.get(), but all other interpreter
> implementation does not leverage ResourcePool at the moment.
>
> We can simply modify PostgreSQL interpreter to push result into
> ResourcePool, so other interpreters, for example SparkInterpreter, can get
> result from ResourcePool and process. I think this implementation will be
> as simple as few lines of code.
>
> Alternative approach is #pr836[2] (hope we can merge this, soon). This
> patch includes the code that automatically pushes all the results into
> ResourcePool, without modification of each interpreter.
>
> Thanks,
> moon
>
> [1]
> https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/resource/ResourcePool.java
> [2] https://github.com/apache/zeppelin/pull/836
>
>
> On Thu, Jun 16, 2016 at 9:13 AM Mohit Jaggi <mo...@gmail.com> wrote:
>
>> I believe z.context() is the only way to share data between interpreters.
>> Within an interpreter data is usually available across paragraphs…perhaps
>> even across notebooks as I guess zeppelin will create a single interpreter
>> in the backend unless you somehow make it use separate ones.
>>
>> > On Jun 16, 2016, at 9:24 PM, Cameron McBride <ca...@gmail.com>
>> wrote:
>> >
>> > Greetings,
>> >
>> > I'm brand new to zeppelin, and this notebook technology looks great.
>> I'm evaluating using it for our data science team, but got it up and
>> running quickly using some PostgreSQL data and some spark tests. The
>> distributed nature of each paragraph, and naturally varying interpreters
>> within a notebook is fantastic. Some of the current built in technology is
>> so refreshing compared with trying to beat in such functionality to other
>> notebook / sharing solutions. Anyhow, I'm stoked and wanted to share that
>> first as part of my first post. :)
>> >
>> > In using the PostgreSQL interpreter, is there a way to share some
>> output data with other paragraphs? I'm trying to run a SQL query, then
>> perhaps do some analysis or derived logic in a different interpreter later
>> in the notebook. Sorry if I missed this somewhere, but my digging couldn't
>> find a workable answer.
>> >
>> > TIA,
>> >
>> > Cameron
>> >
>> >
>>
>>
Re: zeppelin PostgreSQL interpreter data sharing
Posted by moon soo Lee <mo...@apache.org>.
ResourcePool [1] is a data store that all the different types of
interpreter can access.
It allows exchange data between paragraphs/notebooks/interpreters.
However, not all interpreter uses ResourcePool, at the moment.
For example Spark interpreter family provides API to use it (spark,
pyspark, sparkr), which is z.put(), z.get(), but all other interpreter
implementation does not leverage ResourcePool at the moment.
We can simply modify PostgreSQL interpreter to push result into
ResourcePool, so other interpreters, for example SparkInterpreter, can get
result from ResourcePool and process. I think this implementation will be
as simple as few lines of code.
Alternative approach is #pr836[2] (hope we can merge this, soon). This
patch includes the code that automatically pushes all the results into
ResourcePool, without modification of each interpreter.
Thanks,
moon
[1]
https://github.com/apache/zeppelin/blob/master/zeppelin-interpreter/src/main/java/org/apache/zeppelin/resource/ResourcePool.java
[2] https://github.com/apache/zeppelin/pull/836
On Thu, Jun 16, 2016 at 9:13 AM Mohit Jaggi <mo...@gmail.com> wrote:
> I believe z.context() is the only way to share data between interpreters.
> Within an interpreter data is usually available across paragraphs…perhaps
> even across notebooks as I guess zeppelin will create a single interpreter
> in the backend unless you somehow make it use separate ones.
>
> > On Jun 16, 2016, at 9:24 PM, Cameron McBride <ca...@gmail.com>
> wrote:
> >
> > Greetings,
> >
> > I'm brand new to zeppelin, and this notebook technology looks great. I'm
> evaluating using it for our data science team, but got it up and running
> quickly using some PostgreSQL data and some spark tests. The distributed
> nature of each paragraph, and naturally varying interpreters within a
> notebook is fantastic. Some of the current built in technology is so
> refreshing compared with trying to beat in such functionality to other
> notebook / sharing solutions. Anyhow, I'm stoked and wanted to share that
> first as part of my first post. :)
> >
> > In using the PostgreSQL interpreter, is there a way to share some output
> data with other paragraphs? I'm trying to run a SQL query, then perhaps do
> some analysis or derived logic in a different interpreter later in the
> notebook. Sorry if I missed this somewhere, but my digging couldn't find a
> workable answer.
> >
> > TIA,
> >
> > Cameron
> >
> >
>
>
Re: zeppelin PostgreSQL interpreter data sharing
Posted by Mohit Jaggi <mo...@gmail.com>.
I believe z.context() is the only way to share data between interpreters. Within an interpreter data is usually available across paragraphs…perhaps even across notebooks as I guess zeppelin will create a single interpreter in the backend unless you somehow make it use separate ones.
> On Jun 16, 2016, at 9:24 PM, Cameron McBride <ca...@gmail.com> wrote:
>
> Greetings,
>
> I'm brand new to zeppelin, and this notebook technology looks great. I'm evaluating using it for our data science team, but got it up and running quickly using some PostgreSQL data and some spark tests. The distributed nature of each paragraph, and naturally varying interpreters within a notebook is fantastic. Some of the current built in technology is so refreshing compared with trying to beat in such functionality to other notebook / sharing solutions. Anyhow, I'm stoked and wanted to share that first as part of my first post. :)
>
> In using the PostgreSQL interpreter, is there a way to share some output data with other paragraphs? I'm trying to run a SQL query, then perhaps do some analysis or derived logic in a different interpreter later in the notebook. Sorry if I missed this somewhere, but my digging couldn't find a workable answer.
>
> TIA,
>
> Cameron
>
>