Re: [DISCUSS] Integration with Apache Ignite

[Dmitriy Setrakyan <ds...@apache.org>]

Added Ignite dev list to the CC. My comments below...

On Thu, Apr 16, 2015 at 12:37 AM, moon soo Lee <mo...@apache.org> wrote:

> Hello. Welcome to Zeppelin dev list.
>
> In fact, Zeppelin's backend side is mostly written in Java. Therefore
> fundamentally Zeppelin has better compatibility with Java based software.
> And yes, for Ignite SQL it won't be that difficult to integrate.
>

Perhaps we should start from SQL then. Ignite supports JDBC or direct APIs
for SQL. I presume JDBC will be the easiest, right?


>
> For Api based one, like compute, It really depends on how ignite wanted to
> be integrated. if you expect interactive code compile and run, it might
> need two things in general.
>

I was actually hoping that another component would be Streaming (after SQL
is done). For example, you can configure sliding windows in Ignite and then
use standard SQL to query them.


>  One is interactive code compiler. So any codes - like you defining your
> class inside of Zeppelin - should compiled on the fly, line by line, and
> generates class files. I think Scala or Groovy compiler and many other
> language that they have REPL can do this, but not sure about Java.
> Second thing is, generated class files need be somehow transferred and
> loaded to remote cluster nodes.


> So i was thought integration code compiler and remote class loading
> mechanism either Zeppelin side or Ignite side.
>

Actually class loading may not be an issue, because Ignite has a
distributed peer-class-loader which automatically deploys and undeploys
classes.


>
> What do you think?


Sounds exciting. SQL is probably the place to start and we can grow from
there.


>


> Thanks,
> moon
>
>
> On Wed, Apr 15, 2015 at 6:10 PM Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
> > This is my first email to the Zeppelin dev list, so hello everyone.
> >
> > I am on the PPMC of Apache Ignite and I would like to start a discussion
> on
> > integrating Zeppelin with Apache Ignite. Apache Ignite has a fairly
> feature
> > rich family of in-memory components, including distributed caching & data
> > grid, compute grid, streaming & CEP, etc. It is a JAVA-based product, so
> > before continuing the discussion, I would like to find out if Zeppelin
> can
> > support Java (not Scala) runtime. If it can, I think there is definitely
> > room for Ignite SQL on Zeppelin, as well as streaming and compute.
> >
> > Thoughts?
> >
> > D.
> >
>

Re: [DISCUSS] Integration with Apache Ignite

[IT CTO <go...@gmail.com>]

I am not the expert here but I would say that if one wants to integrate
between Zeppelin and Ignite on the SQL level he should implement an Ignite
interpreter. if you take the hive interpreter or the Tajo (still in Dev)
interpreter as an example then you should be able to complete the task...
(BUT as I said I am just a reader on these threads, never did it myself :-)

Eran

On Sun, Apr 19, 2015 at 4:35 PM Dmitriy Setrakyan <ds...@apache.org>
wrote:

> By the way, Ignite implements read-only JDBC driver for select queries,
> will this help?
>
> Also, what would be the right way to take the next step? Should we create a
> Jira ticket?
>
> D.
>
> On Thu, Apr 16, 2015 at 12:52 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> wrote:
>
> > Added Ignite dev list to the CC. My comments below...
> >
> > On Thu, Apr 16, 2015 at 12:37 AM, moon soo Lee <mo...@apache.org> wrote:
> >
> >> Hello. Welcome to Zeppelin dev list.
> >>
> >> In fact, Zeppelin's backend side is mostly written in Java. Therefore
> >> fundamentally Zeppelin has better compatibility with Java based
> software.
> >> And yes, for Ignite SQL it won't be that difficult to integrate.
> >>
> >
> > Perhaps we should start from SQL then. Ignite supports JDBC or direct
> APIs
> > for SQL. I presume JDBC will be the easiest, right?
> >
> >
> >>
> >> For Api based one, like compute, It really depends on how ignite wanted
> to
> >> be integrated. if you expect interactive code compile and run, it might
> >> need two things in general.
> >>
> >
> > I was actually hoping that another component would be Streaming (after
> SQL
> > is done). For example, you can configure sliding windows in Ignite and
> then
> > use standard SQL to query them.
> >
> >
> >>  One is interactive code compiler. So any codes - like you defining your
> >> class inside of Zeppelin - should compiled on the fly, line by line, and
> >> generates class files. I think Scala or Groovy compiler and many other
> >> language that they have REPL can do this, but not sure about Java.
> >> Second thing is, generated class files need be somehow transferred and
> >> loaded to remote cluster nodes.
> >
> >
> >> So i was thought integration code compiler and remote class loading
> >> mechanism either Zeppelin side or Ignite side.
> >>
> >
> > Actually class loading may not be an issue, because Ignite has a
> > distributed peer-class-loader which automatically deploys and undeploys
> > classes.
> >
> >
> >>
> >> What do you think?
> >
> >
> > Sounds exciting. SQL is probably the place to start and we can grow from
> > there.
> >
> >
> >>
> >
> >
> >> Thanks,
> >> moon
> >>
> >>
> >> On Wed, Apr 15, 2015 at 6:10 PM Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> >> wrote:
> >>
> >> > This is my first email to the Zeppelin dev list, so hello everyone.
> >> >
> >> > I am on the PPMC of Apache Ignite and I would like to start a
> >> discussion on
> >> > integrating Zeppelin with Apache Ignite. Apache Ignite has a fairly
> >> feature
> >> > rich family of in-memory components, including distributed caching &
> >> data
> >> > grid, compute grid, streaming & CEP, etc. It is a JAVA-based product,
> so
> >> > before continuing the discussion, I would like to find out if Zeppelin
> >> can
> >> > support Java (not Scala) runtime. If it can, I think there is
> definitely
> >> > room for Ignite SQL on Zeppelin, as well as streaming and compute.
> >> >
> >> > Thoughts?
> >> >
> >> > D.
> >> >
> >>
> >
> >
>

Re: [DISCUSS] Integration with Apache Ignite

[IT CTO <go...@gmail.com>]

I am not the expert here but I would say that if one wants to integrate
between Zeppelin and Ignite on the SQL level he should implement an Ignite
interpreter. if you take the hive interpreter or the Tajo (still in Dev)
interpreter as an example then you should be able to complete the task...
(BUT as I said I am just a reader on these threads, never did it myself :-)

Eran

On Sun, Apr 19, 2015 at 4:35 PM Dmitriy Setrakyan <ds...@apache.org>
wrote:

> By the way, Ignite implements read-only JDBC driver for select queries,
> will this help?
>
> Also, what would be the right way to take the next step? Should we create a
> Jira ticket?
>
> D.
>
> On Thu, Apr 16, 2015 at 12:52 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> wrote:
>
> > Added Ignite dev list to the CC. My comments below...
> >
> > On Thu, Apr 16, 2015 at 12:37 AM, moon soo Lee <mo...@apache.org> wrote:
> >
> >> Hello. Welcome to Zeppelin dev list.
> >>
> >> In fact, Zeppelin's backend side is mostly written in Java. Therefore
> >> fundamentally Zeppelin has better compatibility with Java based
> software.
> >> And yes, for Ignite SQL it won't be that difficult to integrate.
> >>
> >
> > Perhaps we should start from SQL then. Ignite supports JDBC or direct
> APIs
> > for SQL. I presume JDBC will be the easiest, right?
> >
> >
> >>
> >> For Api based one, like compute, It really depends on how ignite wanted
> to
> >> be integrated. if you expect interactive code compile and run, it might
> >> need two things in general.
> >>
> >
> > I was actually hoping that another component would be Streaming (after
> SQL
> > is done). For example, you can configure sliding windows in Ignite and
> then
> > use standard SQL to query them.
> >
> >
> >>  One is interactive code compiler. So any codes - like you defining your
> >> class inside of Zeppelin - should compiled on the fly, line by line, and
> >> generates class files. I think Scala or Groovy compiler and many other
> >> language that they have REPL can do this, but not sure about Java.
> >> Second thing is, generated class files need be somehow transferred and
> >> loaded to remote cluster nodes.
> >
> >
> >> So i was thought integration code compiler and remote class loading
> >> mechanism either Zeppelin side or Ignite side.
> >>
> >
> > Actually class loading may not be an issue, because Ignite has a
> > distributed peer-class-loader which automatically deploys and undeploys
> > classes.
> >
> >
> >>
> >> What do you think?
> >
> >
> > Sounds exciting. SQL is probably the place to start and we can grow from
> > there.
> >
> >
> >>
> >
> >
> >> Thanks,
> >> moon
> >>
> >>
> >> On Wed, Apr 15, 2015 at 6:10 PM Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> >> wrote:
> >>
> >> > This is my first email to the Zeppelin dev list, so hello everyone.
> >> >
> >> > I am on the PPMC of Apache Ignite and I would like to start a
> >> discussion on
> >> > integrating Zeppelin with Apache Ignite. Apache Ignite has a fairly
> >> feature
> >> > rich family of in-memory components, including distributed caching &
> >> data
> >> > grid, compute grid, streaming & CEP, etc. It is a JAVA-based product,
> so
> >> > before continuing the discussion, I would like to find out if Zeppelin
> >> can
> >> > support Java (not Scala) runtime. If it can, I think there is
> definitely
> >> > room for Ignite SQL on Zeppelin, as well as streaming and compute.
> >> >
> >> > Thoughts?
> >> >
> >> > D.
> >> >
> >>
> >
> >
>

Re: [DISCUSS] Integration with Apache Ignite

[Dmitriy Setrakyan <ds...@apache.org>]

By the way, Ignite implements read-only JDBC driver for select queries,
will this help?

Also, what would be the right way to take the next step? Should we create a
Jira ticket?

D.

On Thu, Apr 16, 2015 at 12:52 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> Added Ignite dev list to the CC. My comments below...
>
> On Thu, Apr 16, 2015 at 12:37 AM, moon soo Lee <mo...@apache.org> wrote:
>
>> Hello. Welcome to Zeppelin dev list.
>>
>> In fact, Zeppelin's backend side is mostly written in Java. Therefore
>> fundamentally Zeppelin has better compatibility with Java based software.
>> And yes, for Ignite SQL it won't be that difficult to integrate.
>>
>
> Perhaps we should start from SQL then. Ignite supports JDBC or direct APIs
> for SQL. I presume JDBC will be the easiest, right?
>
>
>>
>> For Api based one, like compute, It really depends on how ignite wanted to
>> be integrated. if you expect interactive code compile and run, it might
>> need two things in general.
>>
>
> I was actually hoping that another component would be Streaming (after SQL
> is done). For example, you can configure sliding windows in Ignite and then
> use standard SQL to query them.
>
>
>>  One is interactive code compiler. So any codes - like you defining your
>> class inside of Zeppelin - should compiled on the fly, line by line, and
>> generates class files. I think Scala or Groovy compiler and many other
>> language that they have REPL can do this, but not sure about Java.
>> Second thing is, generated class files need be somehow transferred and
>> loaded to remote cluster nodes.
>
>
>> So i was thought integration code compiler and remote class loading
>> mechanism either Zeppelin side or Ignite side.
>>
>
> Actually class loading may not be an issue, because Ignite has a
> distributed peer-class-loader which automatically deploys and undeploys
> classes.
>
>
>>
>> What do you think?
>
>
> Sounds exciting. SQL is probably the place to start and we can grow from
> there.
>
>
>>
>
>
>> Thanks,
>> moon
>>
>>
>> On Wed, Apr 15, 2015 at 6:10 PM Dmitriy Setrakyan <ds...@apache.org>
>> wrote:
>>
>> > This is my first email to the Zeppelin dev list, so hello everyone.
>> >
>> > I am on the PPMC of Apache Ignite and I would like to start a
>> discussion on
>> > integrating Zeppelin with Apache Ignite. Apache Ignite has a fairly
>> feature
>> > rich family of in-memory components, including distributed caching &
>> data
>> > grid, compute grid, streaming & CEP, etc. It is a JAVA-based product, so
>> > before continuing the discussion, I would like to find out if Zeppelin
>> can
>> > support Java (not Scala) runtime. If it can, I think there is definitely
>> > room for Ignite SQL on Zeppelin, as well as streaming and compute.
>> >
>> > Thoughts?
>> >
>> > D.
>> >
>>
>
>

Re: [DISCUSS] Integration with Apache Ignite

[Dmitriy Setrakyan <ds...@apache.org>]

By the way, Ignite implements read-only JDBC driver for select queries,
will this help?

Also, what would be the right way to take the next step? Should we create a
Jira ticket?

D.

On Thu, Apr 16, 2015 at 12:52 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

> Added Ignite dev list to the CC. My comments below...
>
> On Thu, Apr 16, 2015 at 12:37 AM, moon soo Lee <mo...@apache.org> wrote:
>
>> Hello. Welcome to Zeppelin dev list.
>>
>> In fact, Zeppelin's backend side is mostly written in Java. Therefore
>> fundamentally Zeppelin has better compatibility with Java based software.
>> And yes, for Ignite SQL it won't be that difficult to integrate.
>>
>
> Perhaps we should start from SQL then. Ignite supports JDBC or direct APIs
> for SQL. I presume JDBC will be the easiest, right?
>
>
>>
>> For Api based one, like compute, It really depends on how ignite wanted to
>> be integrated. if you expect interactive code compile and run, it might
>> need two things in general.
>>
>
> I was actually hoping that another component would be Streaming (after SQL
> is done). For example, you can configure sliding windows in Ignite and then
> use standard SQL to query them.
>
>
>>  One is interactive code compiler. So any codes - like you defining your
>> class inside of Zeppelin - should compiled on the fly, line by line, and
>> generates class files. I think Scala or Groovy compiler and many other
>> language that they have REPL can do this, but not sure about Java.
>> Second thing is, generated class files need be somehow transferred and
>> loaded to remote cluster nodes.
>
>
>> So i was thought integration code compiler and remote class loading
>> mechanism either Zeppelin side or Ignite side.
>>
>
> Actually class loading may not be an issue, because Ignite has a
> distributed peer-class-loader which automatically deploys and undeploys
> classes.
>
>
>>
>> What do you think?
>
>
> Sounds exciting. SQL is probably the place to start and we can grow from
> there.
>
>
>>
>
>
>> Thanks,
>> moon
>>
>>
>> On Wed, Apr 15, 2015 at 6:10 PM Dmitriy Setrakyan <ds...@apache.org>
>> wrote:
>>
>> > This is my first email to the Zeppelin dev list, so hello everyone.
>> >
>> > I am on the PPMC of Apache Ignite and I would like to start a
>> discussion on
>> > integrating Zeppelin with Apache Ignite. Apache Ignite has a fairly
>> feature
>> > rich family of in-memory components, including distributed caching &
>> data
>> > grid, compute grid, streaming & CEP, etc. It is a JAVA-based product, so
>> > before continuing the discussion, I would like to find out if Zeppelin
>> can
>> > support Java (not Scala) runtime. If it can, I think there is definitely
>> > room for Ignite SQL on Zeppelin, as well as streaming and compute.
>> >
>> > Thoughts?
>> >
>> > D.
>> >
>>
>
>