You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@toree.apache.org by Luciano Resende <lu...@gmail.com> on 2017/01/31 16:29:12 UTC
Toree client programming model
The Toree client seems to be a client utility to enable other applications
to interact with Toree/Spark. These applications are not based or aware of
the ZeroMQ protocol, but nevertheless, we are imposing them on a very
similar programming model while evaluating code using the client.
See an example below, where the application programmer has to handle
different options, where more than one option can be called per eval
result, and the order cannot be known prior to the execution time :
val exRes: DeferredExecution = client.execute(code)
.onResult(executeResult => {
...
}).onError(executeReplyError =>{
...
}).onSuccess(executeReplyOk => {
...
}).onStream(streamResult => {
...
})
I was wondering what are the community thoughts on enhancing the client to
abstract its programming model, and having all this abstracted and handled
on the client implementation, and the eval would return one promise (e.g.
onResult which one option would be status success or error) or at most two
promises (e.g. onSuccess and onError).
The only issue I haven't fully understood yet is how streaming is handled
and what are the implication of these changes on this scenario.
Thoughts?
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: Toree client programming model
Posted by Corey Stubbs <ca...@gmail.com>.
That scenario is handled in my above sample API, maybe with a better
interface of a Future/Promise result instead of the option. The client
library would handle waiting for execution to be completed as well as
waiting for the results to be returned. I feel like I am not understanding
your point, I am just getting over a cold so maybe I need to let it sink in
for a bit before it hits me.
The streaming results have nothing to do with Spark Streaming. They are
just messages sent via the output stream, and there can be 0 or more of
these per execution request. The stream results can also continue after
execution has completed (in the event the prints happen in a new thread).
Just to provide examples of what clients currently exist and to see how
they differ from the Toree Client:
Python Client:
https://jupyter-client.readthedocs.io/en/latest/messaging.html#python-api
JavaScript Client:
http://jupyterlab.github.io/services/interfaces/kernel.ikernel.html
On Thu, Feb 2, 2017 at 9:45 AM Luciano Resende <lu...@gmail.com> wrote:
> Let's start by identifying some scenarios that the client is used:
>
> - Users trying to build an application which needs to interactively
> integrate with Spark, where the client is used to submit a piece of code
> and expects an indication that the full results are back to continue doing
> what it needs to do. In this scenario, the client is outside the Jupyter
> realm and should be shielded from any protocol details.
>
> Regarding onStream, is that used on the scope of Spark Streaming, or just
> results (big or small) being sent via output stream? And how do we know
> when the full results have been received by the client side?
>
>
> On Thu, Feb 2, 2017 at 7:35 AM, Corey Stubbs <ca...@gmail.com> wrote:
>
> > So the client programming model is meant to reflect the Jupyter Protocol,
> > which has taken into account issues like streaming. However, the client
> > could still be cleaned up. Maybe something along the lines of:
> >
> > val exRes: DeferredExecution = client.execute(code)
> > .onError(executeReplyError =>{
> > ...
> > }).onSuccess(Option(executeResult) => {
> > ...
> > }).onStream(streamResult => {
> > ...
> > })
> >
> > Key thing being the optional execute result as a parameter to onSuccess.
> > This would be a better interface to handle when there are/are not results
> > from an execution.
> >
> > Kind Regards,
> > Corey Stubbs
> >
> > On Tue, Jan 31, 2017 at 10:29 AM Luciano Resende <lu...@gmail.com>
> > wrote:
> >
> > The Toree client seems to be a client utility to enable other
> applications
> > to interact with Toree/Spark. These applications are not based or aware
> of
> > the ZeroMQ protocol, but nevertheless, we are imposing them on a very
> > similar programming model while evaluating code using the client.
> >
> > See an example below, where the application programmer has to handle
> > different options, where more than one option can be called per eval
> > result, and the order cannot be known prior to the execution time :
> >
> > val exRes: DeferredExecution = client.execute(code)
> > .onResult(executeResult => {
> > ...
> > }).onError(executeReplyError =>{
> > ...
> > }).onSuccess(executeReplyOk => {
> > ...
> > }).onStream(streamResult => {
> > ...
> > })
> >
> >
> > I was wondering what are the community thoughts on enhancing the client
> to
> > abstract its programming model, and having all this abstracted and
> handled
> > on the client implementation, and the eval would return one promise (e.g.
> > onResult which one option would be status success or error) or at most
> two
> > promises (e.g. onSuccess and onError).
> >
> > The only issue I haven't fully understood yet is how streaming is handled
> > and what are the implication of these changes on this scenario.
> >
> > Thoughts?
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
Re: Toree client programming model
Posted by Luciano Resende <lu...@gmail.com>.
Let's start by identifying some scenarios that the client is used:
- Users trying to build an application which needs to interactively
integrate with Spark, where the client is used to submit a piece of code
and expects an indication that the full results are back to continue doing
what it needs to do. In this scenario, the client is outside the Jupyter
realm and should be shielded from any protocol details.
Regarding onStream, is that used on the scope of Spark Streaming, or just
results (big or small) being sent via output stream? And how do we know
when the full results have been received by the client side?
On Thu, Feb 2, 2017 at 7:35 AM, Corey Stubbs <ca...@gmail.com> wrote:
> So the client programming model is meant to reflect the Jupyter Protocol,
> which has taken into account issues like streaming. However, the client
> could still be cleaned up. Maybe something along the lines of:
>
> val exRes: DeferredExecution = client.execute(code)
> .onError(executeReplyError =>{
> ...
> }).onSuccess(Option(executeResult) => {
> ...
> }).onStream(streamResult => {
> ...
> })
>
> Key thing being the optional execute result as a parameter to onSuccess.
> This would be a better interface to handle when there are/are not results
> from an execution.
>
> Kind Regards,
> Corey Stubbs
>
> On Tue, Jan 31, 2017 at 10:29 AM Luciano Resende <lu...@gmail.com>
> wrote:
>
> The Toree client seems to be a client utility to enable other applications
> to interact with Toree/Spark. These applications are not based or aware of
> the ZeroMQ protocol, but nevertheless, we are imposing them on a very
> similar programming model while evaluating code using the client.
>
> See an example below, where the application programmer has to handle
> different options, where more than one option can be called per eval
> result, and the order cannot be known prior to the execution time :
>
> val exRes: DeferredExecution = client.execute(code)
> .onResult(executeResult => {
> ...
> }).onError(executeReplyError =>{
> ...
> }).onSuccess(executeReplyOk => {
> ...
> }).onStream(streamResult => {
> ...
> })
>
>
> I was wondering what are the community thoughts on enhancing the client to
> abstract its programming model, and having all this abstracted and handled
> on the client implementation, and the eval would return one promise (e.g.
> onResult which one option would be status success or error) or at most two
> promises (e.g. onSuccess and onError).
>
> The only issue I haven't fully understood yet is how streaming is handled
> and what are the implication of these changes on this scenario.
>
> Thoughts?
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: Toree client programming model
Posted by Corey Stubbs <ca...@gmail.com>.
So the client programming model is meant to reflect the Jupyter Protocol,
which has taken into account issues like streaming. However, the client
could still be cleaned up. Maybe something along the lines of:
val exRes: DeferredExecution = client.execute(code)
.onError(executeReplyError =>{
...
}).onSuccess(Option(executeResult) => {
...
}).onStream(streamResult => {
...
})
Key thing being the optional execute result as a parameter to onSuccess.
This would be a better interface to handle when there are/are not results
from an execution.
Kind Regards,
Corey Stubbs
On Tue, Jan 31, 2017 at 10:29 AM Luciano Resende <lu...@gmail.com>
wrote:
The Toree client seems to be a client utility to enable other applications
to interact with Toree/Spark. These applications are not based or aware of
the ZeroMQ protocol, but nevertheless, we are imposing them on a very
similar programming model while evaluating code using the client.
See an example below, where the application programmer has to handle
different options, where more than one option can be called per eval
result, and the order cannot be known prior to the execution time :
val exRes: DeferredExecution = client.execute(code)
.onResult(executeResult => {
...
}).onError(executeReplyError =>{
...
}).onSuccess(executeReplyOk => {
...
}).onStream(streamResult => {
...
})
I was wondering what are the community thoughts on enhancing the client to
abstract its programming model, and having all this abstracted and handled
on the client implementation, and the eval would return one promise (e.g.
onResult which one option would be status success or error) or at most two
promises (e.g. onSuccess and onError).
The only issue I haven't fully understood yet is how streaming is handled
and what are the implication of these changes on this scenario.
Thoughts?
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/