You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@heron.apache.org by Saikat Kanjilal <sx...@hotmail.com> on 2017/10/30 01:17:23 UTC
Hello heron community
Hello Folks,
I'm interested in taking and driving the following github component (this means design/development/architecture and more).
https://github.com/twitter/heron/issues/668
[https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 ...<https://github.com/twitter/heron/issues/668>
github.com
heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter
I've joined the heron slack channel as recommended and am ready to start by creating a master JIRA issue to track this if not already done, I wanted some guidance from a committer or two to help guide me a bit take care of at he administrative details and how to get this kicked off the ground. My background includes working on apache reef codebase as well as dabbling in apache mahout and apache flume as well as storm and various nosql/graph databases. Looking forward to being part of the community.
The rough plan of action in my mind so you know:
1) Review heron architecture and codebase in detail
2) Create an umbrella JIRA with the first series of tasks
3) Come up with a design doc
4) Build the interfaces
5) Add more dev tasks to JIRA
6) Drive the implementation
Thanks in advance for your help.
Re: Hello heron community
Posted by Sijie Guo <gu...@gmail.com>.
I have a few thoughts about the scala API and made a comment in your pull
request, purely from a scala user perspective. It would be good if the
scala api can have those scala syntactic sugar.
The comments are:
- function: it would be good to support scala function, so a scala user can
use lambda expression. also it would be good to support java function in
scala api as well. because if there are functions rewritten in java
function, they can be reused in a scala heron app.
- key/value: it would be good to support tuple *(K, V)*. it makes
transformation code much clean in a scala app when transforming between a
streamlet and a kv streamlet.
- package name: I saw the package name is same as java api. it would be
good to distinguish them.
- Sijie
On Tue, Oct 31, 2017 at 2:54 PM, Saikat Kanjilal <sx...@hotmail.com>
wrote:
> Here's an initial PR showing the scala traits that I got as a result of
> running the javatoscala converter on the streamlets API, please take a look
> and let me know if you have any feedback, next steps include putting these
> on the design doc and then starting work on the implementations. To
> reiterate the work was done against a child branch in my fork of heron.
>
>
> https://github.com/skanjila/heron/pull/1
>
>
> Thanks
>
>
> ________________________________
> From: Saikat Kanjilal <sx...@hotmail.com>
> Sent: Monday, October 30, 2017 9:01 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
>
> Thanks Ashwin and yes I am aware of this from reading the documentation.
> I have actually written a little component in reef that runs a simple reef
> job using spark running on yarn, will be curious to look at this code when
> I have some more time.
>
>
> ________________________________
> From: aas.ojj@gmail.com <aa...@gmail.com> on behalf of Ashvin A <
> aasojj@gmail.com>
> Sent: Monday, October 30, 2017 11:36 AM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> Hi Saikat,
>
> Welcome !
>
> Since you are familiar with REEF, you might want to know that the YARN port
> of Heron is based on REEF. You can find some details here:
> https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/
> .
> Heron Documentation - Apache Hadoop YARN Cluster ...<
> https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/
> >
> twitter.github.io
> In addition to out-of-the-box schedulers for Aurora, Heron can also be
> deployed on a YARN cluster with the YARN scheduler. The YARN scheduler is
> implemented using the ...
>
>
>
>
> -ashvin
>
> On Sun, Oct 29, 2017 at 10:19 PM, Karthik Ramasamy <kr...@gmail.com>
> wrote:
>
> > Saikat -
> >
> > Welcome to the Heron Community. It will be great to see support for Scale
> > API. Current API are of two forms -
> >
> > - procedural/object oriented api (this is very low level where you have
> to
> > write the DAG operators and assemble them) - somewhat similar to Storm
> > - functional (this is more higher level that use functions like map,
> > flatMap, etc)
> >
> > Here are few suggestions and recommendations.
> >
> > - Join the slack for any quick questions and answers (which you have
> > already done)
> >
> > - We use Github to track issues. In your case, it might be a easier to
> > track using projects - for your convenience
> > I have created a project called Scala API for Heron -
> > https://github.com/twitter/heron/projects/3 <https://github.com/twitter/
> [https://avatars1.githubusercontent.com/u/50278?s=400&v=4]<
> https://github.com/twitter/heron/projects/3>
>
> twitter/heron<https://github.com/twitter/heron/projects/3>
> github.com
> heron - Heron is a realtime, distributed, fault-tolerant stream processing
> engine from Twitter
>
>
>
> > heron/projects/3>
> >
> > - It will be great to have a design doc, adding this as a first task into
> > backlog. Once the design doc is complete, you can post it for review and
> > comments
> > for the community. Google doc is an easy way to share and get feedback
> > unless there is other preference.
> >
> > - Once review is complete, you can create additional tasks and issues and
> > link them to the project.
> >
> > Hope this helps to jump start.
> >
> > cheers
> > /karthik
> >
> > > On Oct 29, 2017, at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
> > wrote:
> > >
> > > Hello Folks,
> > >
> > > I'm interested in taking and driving the following github component
> > (this means design/development/architecture and more).
> > >
> > >
> > > https://github.com/twitter/heron/issues/668
> > >
> > >
> > > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies · Issue #668 ...<
> > https://github.com/twitter/heron/issues/668>
> > > github.com
> > > heron - Heron is a realtime, distributed, fault-tolerant stream
> > processing engine from Twitter
> > >
> > >
> > >
> > >
> > > I've joined the heron slack channel as recommended and am ready to
> start
> > by creating a master JIRA issue to track this if not already done, I
> wanted
> > some guidance from a committer or two to help guide me a bit take care of
> > at he administrative details and how to get this kicked off the ground.
> My
> > background includes working on apache reef codebase as well as dabbling
> in
> > apache mahout and apache flume as well as storm and various nosql/graph
> > databases. Looking forward to being part of the community.
> > >
> > >
> > >
> > > The rough plan of action in my mind so you know:
> > >
> > > 1) Review heron architecture and codebase in detail
> > >
> > > 2) Create an umbrella JIRA with the first series of tasks
> > >
> > > 3) Come up with a design doc
> > >
> > > 4) Build the interfaces
> > >
> > > 5) Add more dev tasks to JIRA
> > >
> > > 6) Drive the implementation
> > >
> > >
> > > Thanks in advance for your help.
> > >
> >
> >
>
Re: Hello heron community
Posted by Saikat Kanjilal <sx...@hotmail.com>.
Here's an initial PR showing the scala traits that I got as a result of running the javatoscala converter on the streamlets API, please take a look and let me know if you have any feedback, next steps include putting these on the design doc and then starting work on the implementations. To reiterate the work was done against a child branch in my fork of heron.
https://github.com/skanjila/heron/pull/1
Thanks
________________________________
From: Saikat Kanjilal <sx...@hotmail.com>
Sent: Monday, October 30, 2017 9:01 PM
To: dev@heron.incubator.apache.org
Subject: Re: Hello heron community
Thanks Ashwin and yes I am aware of this from reading the documentation. I have actually written a little component in reef that runs a simple reef job using spark running on yarn, will be curious to look at this code when I have some more time.
________________________________
From: aas.ojj@gmail.com <aa...@gmail.com> on behalf of Ashvin A <aa...@gmail.com>
Sent: Monday, October 30, 2017 11:36 AM
To: dev@heron.incubator.apache.org
Subject: Re: Hello heron community
Hi Saikat,
Welcome !
Since you are familiar with REEF, you might want to know that the YARN port
of Heron is based on REEF. You can find some details here:
https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/.
Heron Documentation - Apache Hadoop YARN Cluster ...<https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/>
twitter.github.io
In addition to out-of-the-box schedulers for Aurora, Heron can also be deployed on a YARN cluster with the YARN scheduler. The YARN scheduler is implemented using the ...
-ashvin
On Sun, Oct 29, 2017 at 10:19 PM, Karthik Ramasamy <kr...@gmail.com>
wrote:
> Saikat -
>
> Welcome to the Heron Community. It will be great to see support for Scale
> API. Current API are of two forms -
>
> - procedural/object oriented api (this is very low level where you have to
> write the DAG operators and assemble them) - somewhat similar to Storm
> - functional (this is more higher level that use functions like map,
> flatMap, etc)
>
> Here are few suggestions and recommendations.
>
> - Join the slack for any quick questions and answers (which you have
> already done)
>
> - We use Github to track issues. In your case, it might be a easier to
> track using projects - for your convenience
> I have created a project called Scala API for Heron -
> https://github.com/twitter/heron/projects/3 <https://github.com/twitter/
[https://avatars1.githubusercontent.com/u/50278?s=400&v=4]<https://github.com/twitter/heron/projects/3>
twitter/heron<https://github.com/twitter/heron/projects/3>
github.com
heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter
> heron/projects/3>
>
> - It will be great to have a design doc, adding this as a first task into
> backlog. Once the design doc is complete, you can post it for review and
> comments
> for the community. Google doc is an easy way to share and get feedback
> unless there is other preference.
>
> - Once review is complete, you can create additional tasks and issues and
> link them to the project.
>
> Hope this helps to jump start.
>
> cheers
> /karthik
>
> > On Oct 29, 2017, at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
> wrote:
> >
> > Hello Folks,
> >
> > I'm interested in taking and driving the following github component
> (this means design/development/architecture and more).
> >
> >
> > https://github.com/twitter/heron/issues/668
> >
> >
> > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 ...<
> https://github.com/twitter/heron/issues/668>
> > github.com
> > heron - Heron is a realtime, distributed, fault-tolerant stream
> processing engine from Twitter
> >
> >
> >
> >
> > I've joined the heron slack channel as recommended and am ready to start
> by creating a master JIRA issue to track this if not already done, I wanted
> some guidance from a committer or two to help guide me a bit take care of
> at he administrative details and how to get this kicked off the ground. My
> background includes working on apache reef codebase as well as dabbling in
> apache mahout and apache flume as well as storm and various nosql/graph
> databases. Looking forward to being part of the community.
> >
> >
> >
> > The rough plan of action in my mind so you know:
> >
> > 1) Review heron architecture and codebase in detail
> >
> > 2) Create an umbrella JIRA with the first series of tasks
> >
> > 3) Come up with a design doc
> >
> > 4) Build the interfaces
> >
> > 5) Add more dev tasks to JIRA
> >
> > 6) Drive the implementation
> >
> >
> > Thanks in advance for your help.
> >
>
>
Re: Hello heron community
Posted by Saikat Kanjilal <sx...@hotmail.com>.
Thanks Ashwin and yes I am aware of this from reading the documentation. I have actually written a little component in reef that runs a simple reef job using spark running on yarn, will be curious to look at this code when I have some more time.
________________________________
From: aas.ojj@gmail.com <aa...@gmail.com> on behalf of Ashvin A <aa...@gmail.com>
Sent: Monday, October 30, 2017 11:36 AM
To: dev@heron.incubator.apache.org
Subject: Re: Hello heron community
Hi Saikat,
Welcome !
Since you are familiar with REEF, you might want to know that the YARN port
of Heron is based on REEF. You can find some details here:
https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/.
Heron Documentation - Apache Hadoop YARN Cluster ...<https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/>
twitter.github.io
In addition to out-of-the-box schedulers for Aurora, Heron can also be deployed on a YARN cluster with the YARN scheduler. The YARN scheduler is implemented using the ...
-ashvin
On Sun, Oct 29, 2017 at 10:19 PM, Karthik Ramasamy <kr...@gmail.com>
wrote:
> Saikat -
>
> Welcome to the Heron Community. It will be great to see support for Scale
> API. Current API are of two forms -
>
> - procedural/object oriented api (this is very low level where you have to
> write the DAG operators and assemble them) - somewhat similar to Storm
> - functional (this is more higher level that use functions like map,
> flatMap, etc)
>
> Here are few suggestions and recommendations.
>
> - Join the slack for any quick questions and answers (which you have
> already done)
>
> - We use Github to track issues. In your case, it might be a easier to
> track using projects - for your convenience
> I have created a project called Scala API for Heron -
> https://github.com/twitter/heron/projects/3 <https://github.com/twitter/
[https://avatars1.githubusercontent.com/u/50278?s=400&v=4]<https://github.com/twitter/heron/projects/3>
twitter/heron<https://github.com/twitter/heron/projects/3>
github.com
heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter
> heron/projects/3>
>
> - It will be great to have a design doc, adding this as a first task into
> backlog. Once the design doc is complete, you can post it for review and
> comments
> for the community. Google doc is an easy way to share and get feedback
> unless there is other preference.
>
> - Once review is complete, you can create additional tasks and issues and
> link them to the project.
>
> Hope this helps to jump start.
>
> cheers
> /karthik
>
> > On Oct 29, 2017, at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
> wrote:
> >
> > Hello Folks,
> >
> > I'm interested in taking and driving the following github component
> (this means design/development/architecture and more).
> >
> >
> > https://github.com/twitter/heron/issues/668
> >
> >
> > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 ...<
> https://github.com/twitter/heron/issues/668>
> > github.com
> > heron - Heron is a realtime, distributed, fault-tolerant stream
> processing engine from Twitter
> >
> >
> >
> >
> > I've joined the heron slack channel as recommended and am ready to start
> by creating a master JIRA issue to track this if not already done, I wanted
> some guidance from a committer or two to help guide me a bit take care of
> at he administrative details and how to get this kicked off the ground. My
> background includes working on apache reef codebase as well as dabbling in
> apache mahout and apache flume as well as storm and various nosql/graph
> databases. Looking forward to being part of the community.
> >
> >
> >
> > The rough plan of action in my mind so you know:
> >
> > 1) Review heron architecture and codebase in detail
> >
> > 2) Create an umbrella JIRA with the first series of tasks
> >
> > 3) Come up with a design doc
> >
> > 4) Build the interfaces
> >
> > 5) Add more dev tasks to JIRA
> >
> > 6) Drive the implementation
> >
> >
> > Thanks in advance for your help.
> >
>
>
Re: Hello heron community
Posted by Ashvin A <aa...@gmail.com>.
Hi Saikat,
Welcome !
Since you are familiar with REEF, you might want to know that the YARN port
of Heron is based on REEF. You can find some details here:
https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/.
-ashvin
On Sun, Oct 29, 2017 at 10:19 PM, Karthik Ramasamy <kr...@gmail.com>
wrote:
> Saikat -
>
> Welcome to the Heron Community. It will be great to see support for Scale
> API. Current API are of two forms -
>
> - procedural/object oriented api (this is very low level where you have to
> write the DAG operators and assemble them) - somewhat similar to Storm
> - functional (this is more higher level that use functions like map,
> flatMap, etc)
>
> Here are few suggestions and recommendations.
>
> - Join the slack for any quick questions and answers (which you have
> already done)
>
> - We use Github to track issues. In your case, it might be a easier to
> track using projects - for your convenience
> I have created a project called Scala API for Heron -
> https://github.com/twitter/heron/projects/3 <https://github.com/twitter/
> heron/projects/3>
>
> - It will be great to have a design doc, adding this as a first task into
> backlog. Once the design doc is complete, you can post it for review and
> comments
> for the community. Google doc is an easy way to share and get feedback
> unless there is other preference.
>
> - Once review is complete, you can create additional tasks and issues and
> link them to the project.
>
> Hope this helps to jump start.
>
> cheers
> /karthik
>
> > On Oct 29, 2017, at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
> wrote:
> >
> > Hello Folks,
> >
> > I'm interested in taking and driving the following github component
> (this means design/development/architecture and more).
> >
> >
> > https://github.com/twitter/heron/issues/668
> >
> >
> > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 ...<
> https://github.com/twitter/heron/issues/668>
> > github.com
> > heron - Heron is a realtime, distributed, fault-tolerant stream
> processing engine from Twitter
> >
> >
> >
> >
> > I've joined the heron slack channel as recommended and am ready to start
> by creating a master JIRA issue to track this if not already done, I wanted
> some guidance from a committer or two to help guide me a bit take care of
> at he administrative details and how to get this kicked off the ground. My
> background includes working on apache reef codebase as well as dabbling in
> apache mahout and apache flume as well as storm and various nosql/graph
> databases. Looking forward to being part of the community.
> >
> >
> >
> > The rough plan of action in my mind so you know:
> >
> > 1) Review heron architecture and codebase in detail
> >
> > 2) Create an umbrella JIRA with the first series of tasks
> >
> > 3) Come up with a design doc
> >
> > 4) Build the interfaces
> >
> > 5) Add more dev tasks to JIRA
> >
> > 6) Drive the implementation
> >
> >
> > Thanks in advance for your help.
> >
>
>
Re: Hello heron community
Posted by Karthik Ramasamy <kr...@gmail.com>.
Saikat -
Welcome to the Heron Community. It will be great to see support for Scale API. Current API are of two forms -
- procedural/object oriented api (this is very low level where you have to write the DAG operators and assemble them) - somewhat similar to Storm
- functional (this is more higher level that use functions like map, flatMap, etc)
Here are few suggestions and recommendations.
- Join the slack for any quick questions and answers (which you have already done)
- We use Github to track issues. In your case, it might be a easier to track using projects - for your convenience
I have created a project called Scala API for Heron - https://github.com/twitter/heron/projects/3 <https://github.com/twitter/heron/projects/3>
- It will be great to have a design doc, adding this as a first task into backlog. Once the design doc is complete, you can post it for review and comments
for the community. Google doc is an easy way to share and get feedback unless there is other preference.
- Once review is complete, you can create additional tasks and issues and link them to the project.
Hope this helps to jump start.
cheers
/karthik
> On Oct 29, 2017, at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com> wrote:
>
> Hello Folks,
>
> I'm interested in taking and driving the following github component (this means design/development/architecture and more).
>
>
> https://github.com/twitter/heron/issues/668
>
>
> [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 ...<https://github.com/twitter/heron/issues/668>
> github.com
> heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter
>
>
>
>
> I've joined the heron slack channel as recommended and am ready to start by creating a master JIRA issue to track this if not already done, I wanted some guidance from a committer or two to help guide me a bit take care of at he administrative details and how to get this kicked off the ground. My background includes working on apache reef codebase as well as dabbling in apache mahout and apache flume as well as storm and various nosql/graph databases. Looking forward to being part of the community.
>
>
>
> The rough plan of action in my mind so you know:
>
> 1) Review heron architecture and codebase in detail
>
> 2) Create an umbrella JIRA with the first series of tasks
>
> 3) Come up with a design doc
>
> 4) Build the interfaces
>
> 5) Add more dev tasks to JIRA
>
> 6) Drive the implementation
>
>
> Thanks in advance for your help.
>
Re: Hello heron community
Posted by Sanjeev Kulkarni <sa...@gmail.com>.
My initial hunch is that these are different goals and therefore concur
with your thought that you could do this in phase 2.
On Mon, Oct 30, 2017 at 3:03 PM, Saikat Kanjilal <sx...@hotmail.com>
wrote:
> The goal is to tie together spark streaming and heron, I could very easily
> see an architecture where we build topologies in heron to apply business
> logic and pump them into time windows with spark streaming where can apply
> machine learning algorithms inside these windows.
>
>
> Detailed Example:
>
>
> Stock Trading or gamin use case:
>
>
> Heron is used to extract real time data from a gaming or trading client,
>
> Heron will apply a set of business rules to cleanup this data to fit into
> a set of machine learning models
>
> We need to then figure out how to connect spark streaming to heron, in the
> case of spark streaming it listens to some tcp or http endpoint so the
> objective would be to figure out the bridge between the two worlds.
>
>
> My point here is that if I am going through and building out a scala API
> it might make sense to also build a connector to a tech stack entirely
> written using scala as a first class citizen.
>
>
>
> Let me know your thoughts, we can also push this to phase 2 once the heron
> scala api is readily available.
>
>
> ________________________________
> From: Sanjeev Kulkarni <sa...@gmail.com>
> Sent: Monday, October 30, 2017 2:54 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> I didn't really get what you meant by 'stream endpoint would be coming from
> heron'. Could you please elaborate? Preferably with an example?
>
> On Mon, Oct 30, 2017 at 2:26 PM, Saikat Kanjilal <sx...@hotmail.com>
> wrote:
>
> > I'm conflicted a bit on 1, here's why I feel like most of our users would
> > want to use the spouts/bolts API who are coming over from Storm or using
> > heron for the first time, however the Streamlets interface seems like it
> > has the right level of abstraction, I think I will start with the
> Streamlet
> > interface and add in the spout/bolt API as needed. One other thought I
> > had was to figure out an integration plan with spark, to this end the fit
> > between heron and spark would be in using spark streaming where the
> stream
> > endpoint would be coming from heron, not sure if that should be part of
> > this effort or not, what do you guys think?
> >
> >
> > ________________________________
> > From: Sanjeev Kulkarni <sa...@gmail.com>
> > Sent: Monday, October 30, 2017 2:17 PM
> > To: dev@heron.incubator.apache.org
> > Subject: Re: Hello heron community
> >
> > I have a couple of questions
> > 1. Do you plan on exposing both the low level(spouts/bolts) and the
> > streamlet api? Or are you preferring one over another?
> > 2. My suggestion would be to start with the Streamlet interface.
> Primarily
> > because a) Most of scala ml libraries that I've seen tend to operate on
> the
> > streamlet kind of interface and b) It has a far smaller surface area(i.e.
> > number of interfaces) so might be easy to get something up quickly for
> > testing.
> > Thanks!
> >
> > On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sx...@hotmail.com>
> > wrote:
> >
> > > Thanks Sanjeev, I had an initial idea that I wanted to float on the
> > list,
> > > I was thinking that as part of the initial scala port of the API I'd
> like
> > > to propose that we use this tool: http://javatoscala.com/
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framework
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> >
> > >
> > > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framework
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> >
> > > javatoscala.com
> > > How does it work? Java to Scala converter is created with Play
> framework
> > > and Scalagen library. I don't want you to see my code. No problem. Java
> > to
> > > Scala converter ...
> > >
> > >
> > > I propose that we use the above tool to do an initial conversion of all
> > > the Java functional and oo interfaces and then plug that into the
> initial
> > > design doc as well as the codebase to get things off the ground. I
> have
> > > already forked the heron repo and will be doing this off of a child
> > branch
> > > based on my fork.
> > >
> > >
> > > How does that sound to folks, anyone has strong objections, of course I
> > > fully realize that I will still need to do a lot of work around
> > refactoring
> > > the code even when using the converter so I'm prepared for that, the
> goal
> > > in using the converters is to save a bit of time so that manual
> > > intervention isnt necessarily needed in designing every interface.
> > >
> > > Thoughts.
> > >
> > > ________________________________
> > > From: Sanjeev Kulkarni <sa...@gmail.com>
> > > Sent: Sunday, October 29, 2017 10:18 PM
> > > To: dev@heron.incubator.apache.org
> > > Subject: Re: Hello heron community
> > >
> > > Hi Saikat,
> > > Welcome to the Heron users group. Your plan of action sounds good to
> me.
> > A
> > > native scala interface would be great for all those scala fans. Eagerly
> > > awaiting design doc.
> > > Thanks!
> > >
> > > On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
> > > wrote:
> > >
> > > > Hello Folks,
> > > >
> > > > I'm interested in taking and driving the following github component
> > (this
> > > > means design/development/architecture and more).
> > > >
> > > >
> > > > https://github.com/twitter/heron/issues/668
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > github.com
> > > Support for Scala Instance and APIs.
> > >
> > >
> > >
> > > >
> > > >
> > > > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> > > > github.com/twitter/heron/issues/668>
> > > >
> > > > Ability to write native scala topologies · Issue #668 ...<
> > > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > github.com
> > > Support for Scala Instance and APIs.
> > >
> > >
> > >
> > > > github.com
> > > > heron - Heron is a realtime, distributed, fault-tolerant stream
> > > processing
> > > > engine from Twitter
> > > >
> > > >
> > > >
> > > >
> > > > I've joined the heron slack channel as recommended and am ready to
> > start
> > > > by creating a master JIRA issue to track this if not already done, I
> > > wanted
> > > > some guidance from a committer or two to help guide me a bit take
> care
> > of
> > > > at he administrative details and how to get this kicked off the
> ground.
> > > My
> > > > background includes working on apache reef codebase as well as
> dabbling
> > > in
> > > > apache mahout and apache flume as well as storm and various
> nosql/graph
> > > > databases. Looking forward to being part of the community.
> > > >
> > > >
> > > >
> > > > The rough plan of action in my mind so you know:
> > > >
> > > > 1) Review heron architecture and codebase in detail
> > > >
> > > > 2) Create an umbrella JIRA with the first series of tasks
> > > >
> > > > 3) Come up with a design doc
> > > >
> > > > 4) Build the interfaces
> > > >
> > > > 5) Add more dev tasks to JIRA
> > > >
> > > > 6) Drive the implementation
> > > >
> > > >
> > > > Thanks in advance for your help.
> > > >
> > > >
> > >
> >
>
Re: Hello heron community
Posted by Saikat Kanjilal <sx...@hotmail.com>.
The goal is to tie together spark streaming and heron, I could very easily see an architecture where we build topologies in heron to apply business logic and pump them into time windows with spark streaming where can apply machine learning algorithms inside these windows.
Detailed Example:
Stock Trading or gamin use case:
Heron is used to extract real time data from a gaming or trading client,
Heron will apply a set of business rules to cleanup this data to fit into a set of machine learning models
We need to then figure out how to connect spark streaming to heron, in the case of spark streaming it listens to some tcp or http endpoint so the objective would be to figure out the bridge between the two worlds.
My point here is that if I am going through and building out a scala API it might make sense to also build a connector to a tech stack entirely written using scala as a first class citizen.
Let me know your thoughts, we can also push this to phase 2 once the heron scala api is readily available.
________________________________
From: Sanjeev Kulkarni <sa...@gmail.com>
Sent: Monday, October 30, 2017 2:54 PM
To: dev@heron.incubator.apache.org
Subject: Re: Hello heron community
I didn't really get what you meant by 'stream endpoint would be coming from
heron'. Could you please elaborate? Preferably with an example?
On Mon, Oct 30, 2017 at 2:26 PM, Saikat Kanjilal <sx...@hotmail.com>
wrote:
> I'm conflicted a bit on 1, here's why I feel like most of our users would
> want to use the spouts/bolts API who are coming over from Storm or using
> heron for the first time, however the Streamlets interface seems like it
> has the right level of abstraction, I think I will start with the Streamlet
> interface and add in the spout/bolt API as needed. One other thought I
> had was to figure out an integration plan with spark, to this end the fit
> between heron and spark would be in using spark streaming where the stream
> endpoint would be coming from heron, not sure if that should be part of
> this effort or not, what do you guys think?
>
>
> ________________________________
> From: Sanjeev Kulkarni <sa...@gmail.com>
> Sent: Monday, October 30, 2017 2:17 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> I have a couple of questions
> 1. Do you plan on exposing both the low level(spouts/bolts) and the
> streamlet api? Or are you preferring one over another?
> 2. My suggestion would be to start with the Streamlet interface. Primarily
> because a) Most of scala ml libraries that I've seen tend to operate on the
> streamlet kind of interface and b) It has a far smaller surface area(i.e.
> number of interfaces) so might be easy to get something up quickly for
> testing.
> Thanks!
>
> On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sx...@hotmail.com>
> wrote:
>
> > Thanks Sanjeev, I had an initial idea that I wanted to float on the
> list,
> > I was thinking that as part of the initial scala port of the API I'd like
> > to propose that we use this tool: http://javatoscala.com/
Java to Scala converter<http://javatoscala.com/>
javatoscala.com
How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ...
> Java to Scala converter<http://javatoscala.com/>
Java to Scala converter<http://javatoscala.com/>
javatoscala.com
How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ...
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> >
> > Java to Scala converter<http://javatoscala.com/>
Java to Scala converter<http://javatoscala.com/>
javatoscala.com
How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ...
> Java to Scala converter<http://javatoscala.com/>
Java to Scala converter<http://javatoscala.com/>
javatoscala.com
How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ...
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framework
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> > I propose that we use the above tool to do an initial conversion of all
> > the Java functional and oo interfaces and then plug that into the initial
> > design doc as well as the codebase to get things off the ground. I have
> > already forked the heron repo and will be doing this off of a child
> branch
> > based on my fork.
> >
> >
> > How does that sound to folks, anyone has strong objections, of course I
> > fully realize that I will still need to do a lot of work around
> refactoring
> > the code even when using the converter so I'm prepared for that, the goal
> > in using the converters is to save a bit of time so that manual
> > intervention isnt necessarily needed in designing every interface.
> >
> > Thoughts.
> >
> > ________________________________
> > From: Sanjeev Kulkarni <sa...@gmail.com>
> > Sent: Sunday, October 29, 2017 10:18 PM
> > To: dev@heron.incubator.apache.org
> > Subject: Re: Hello heron community
> >
> > Hi Saikat,
> > Welcome to the Heron users group. Your plan of action sounds good to me.
> A
> > native scala interface would be great for all those scala fans. Eagerly
> > awaiting design doc.
> > Thanks!
> >
> > On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
> > wrote:
> >
> > > Hello Folks,
> > >
> > > I'm interested in taking and driving the following github component
> (this
> > > means design/development/architecture and more).
> > >
> > >
> > > https://github.com/twitter/heron/issues/668
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > >
> > >
> > > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> > > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies · Issue #668 ...<
> > > https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > github.com
> > > heron - Heron is a realtime, distributed, fault-tolerant stream
> > processing
> > > engine from Twitter
> > >
> > >
> > >
> > >
> > > I've joined the heron slack channel as recommended and am ready to
> start
> > > by creating a master JIRA issue to track this if not already done, I
> > wanted
> > > some guidance from a committer or two to help guide me a bit take care
> of
> > > at he administrative details and how to get this kicked off the ground.
> > My
> > > background includes working on apache reef codebase as well as dabbling
> > in
> > > apache mahout and apache flume as well as storm and various nosql/graph
> > > databases. Looking forward to being part of the community.
> > >
> > >
> > >
> > > The rough plan of action in my mind so you know:
> > >
> > > 1) Review heron architecture and codebase in detail
> > >
> > > 2) Create an umbrella JIRA with the first series of tasks
> > >
> > > 3) Come up with a design doc
> > >
> > > 4) Build the interfaces
> > >
> > > 5) Add more dev tasks to JIRA
> > >
> > > 6) Drive the implementation
> > >
> > >
> > > Thanks in advance for your help.
> > >
> > >
> >
>
Re: Hello heron community
Posted by Sanjeev Kulkarni <sa...@gmail.com>.
I didn't really get what you meant by 'stream endpoint would be coming from
heron'. Could you please elaborate? Preferably with an example?
On Mon, Oct 30, 2017 at 2:26 PM, Saikat Kanjilal <sx...@hotmail.com>
wrote:
> I'm conflicted a bit on 1, here's why I feel like most of our users would
> want to use the spouts/bolts API who are coming over from Storm or using
> heron for the first time, however the Streamlets interface seems like it
> has the right level of abstraction, I think I will start with the Streamlet
> interface and add in the spout/bolt API as needed. One other thought I
> had was to figure out an integration plan with spark, to this end the fit
> between heron and spark would be in using spark streaming where the stream
> endpoint would be coming from heron, not sure if that should be part of
> this effort or not, what do you guys think?
>
>
> ________________________________
> From: Sanjeev Kulkarni <sa...@gmail.com>
> Sent: Monday, October 30, 2017 2:17 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> I have a couple of questions
> 1. Do you plan on exposing both the low level(spouts/bolts) and the
> streamlet api? Or are you preferring one over another?
> 2. My suggestion would be to start with the Streamlet interface. Primarily
> because a) Most of scala ml libraries that I've seen tend to operate on the
> streamlet kind of interface and b) It has a far smaller surface area(i.e.
> number of interfaces) so might be easy to get something up quickly for
> testing.
> Thanks!
>
> On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sx...@hotmail.com>
> wrote:
>
> > Thanks Sanjeev, I had an initial idea that I wanted to float on the
> list,
> > I was thinking that as part of the initial scala port of the API I'd like
> > to propose that we use this tool: http://javatoscala.com/
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> >
> > Java to Scala converter<http://javatoscala.com/>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
>
> > javatoscala.com
> > How does it work? Java to Scala converter is created with Play framework
> > and Scalagen library. I don't want you to see my code. No problem. Java
> to
> > Scala converter ...
> >
> >
> > I propose that we use the above tool to do an initial conversion of all
> > the Java functional and oo interfaces and then plug that into the initial
> > design doc as well as the codebase to get things off the ground. I have
> > already forked the heron repo and will be doing this off of a child
> branch
> > based on my fork.
> >
> >
> > How does that sound to folks, anyone has strong objections, of course I
> > fully realize that I will still need to do a lot of work around
> refactoring
> > the code even when using the converter so I'm prepared for that, the goal
> > in using the converters is to save a bit of time so that manual
> > intervention isnt necessarily needed in designing every interface.
> >
> > Thoughts.
> >
> > ________________________________
> > From: Sanjeev Kulkarni <sa...@gmail.com>
> > Sent: Sunday, October 29, 2017 10:18 PM
> > To: dev@heron.incubator.apache.org
> > Subject: Re: Hello heron community
> >
> > Hi Saikat,
> > Welcome to the Heron users group. Your plan of action sounds good to me.
> A
> > native scala interface would be great for all those scala fans. Eagerly
> > awaiting design doc.
> > Thanks!
> >
> > On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
> > wrote:
> >
> > > Hello Folks,
> > >
> > > I'm interested in taking and driving the following github component
> (this
> > > means design/development/architecture and more).
> > >
> > >
> > > https://github.com/twitter/heron/issues/668
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > >
> > >
> > > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> > > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies · Issue #668 ...<
> > > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 · twitter/heron<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > Support for Scala Instance and APIs.
> >
> >
> >
> > > github.com
> > > heron - Heron is a realtime, distributed, fault-tolerant stream
> > processing
> > > engine from Twitter
> > >
> > >
> > >
> > >
> > > I've joined the heron slack channel as recommended and am ready to
> start
> > > by creating a master JIRA issue to track this if not already done, I
> > wanted
> > > some guidance from a committer or two to help guide me a bit take care
> of
> > > at he administrative details and how to get this kicked off the ground.
> > My
> > > background includes working on apache reef codebase as well as dabbling
> > in
> > > apache mahout and apache flume as well as storm and various nosql/graph
> > > databases. Looking forward to being part of the community.
> > >
> > >
> > >
> > > The rough plan of action in my mind so you know:
> > >
> > > 1) Review heron architecture and codebase in detail
> > >
> > > 2) Create an umbrella JIRA with the first series of tasks
> > >
> > > 3) Come up with a design doc
> > >
> > > 4) Build the interfaces
> > >
> > > 5) Add more dev tasks to JIRA
> > >
> > > 6) Drive the implementation
> > >
> > >
> > > Thanks in advance for your help.
> > >
> > >
> >
>
Re: Hello heron community
Posted by Saikat Kanjilal <sx...@hotmail.com>.
I'm conflicted a bit on 1, here's why I feel like most of our users would want to use the spouts/bolts API who are coming over from Storm or using heron for the first time, however the Streamlets interface seems like it has the right level of abstraction, I think I will start with the Streamlet interface and add in the spout/bolt API as needed. One other thought I had was to figure out an integration plan with spark, to this end the fit between heron and spark would be in using spark streaming where the stream endpoint would be coming from heron, not sure if that should be part of this effort or not, what do you guys think?
________________________________
From: Sanjeev Kulkarni <sa...@gmail.com>
Sent: Monday, October 30, 2017 2:17 PM
To: dev@heron.incubator.apache.org
Subject: Re: Hello heron community
I have a couple of questions
1. Do you plan on exposing both the low level(spouts/bolts) and the
streamlet api? Or are you preferring one over another?
2. My suggestion would be to start with the Streamlet interface. Primarily
because a) Most of scala ml libraries that I've seen tend to operate on the
streamlet kind of interface and b) It has a far smaller surface area(i.e.
number of interfaces) so might be easy to get something up quickly for
testing.
Thanks!
On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sx...@hotmail.com>
wrote:
> Thanks Sanjeev, I had an initial idea that I wanted to float on the list,
> I was thinking that as part of the initial scala port of the API I'd like
> to propose that we use this tool: http://javatoscala.com/
Java to Scala converter<http://javatoscala.com/>
javatoscala.com
How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ...
>
> Java to Scala converter<http://javatoscala.com/>
Java to Scala converter<http://javatoscala.com/>
javatoscala.com
How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ...
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
> I propose that we use the above tool to do an initial conversion of all
> the Java functional and oo interfaces and then plug that into the initial
> design doc as well as the codebase to get things off the ground. I have
> already forked the heron repo and will be doing this off of a child branch
> based on my fork.
>
>
> How does that sound to folks, anyone has strong objections, of course I
> fully realize that I will still need to do a lot of work around refactoring
> the code even when using the converter so I'm prepared for that, the goal
> in using the converters is to save a bit of time so that manual
> intervention isnt necessarily needed in designing every interface.
>
> Thoughts.
>
> ________________________________
> From: Sanjeev Kulkarni <sa...@gmail.com>
> Sent: Sunday, October 29, 2017 10:18 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> Hi Saikat,
> Welcome to the Heron users group. Your plan of action sounds good to me. A
> native scala interface would be great for all those scala fans. Eagerly
> awaiting design doc.
> Thanks!
>
> On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
> wrote:
>
> > Hello Folks,
> >
> > I'm interested in taking and driving the following github component (this
> > means design/development/architecture and more).
> >
> >
> > https://github.com/twitter/heron/issues/668
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> github.com
> Support for Scala Instance and APIs.
>
>
>
> >
> >
> > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 ...<
> > https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > heron - Heron is a realtime, distributed, fault-tolerant stream
> processing
> > engine from Twitter
> >
> >
> >
> >
> > I've joined the heron slack channel as recommended and am ready to start
> > by creating a master JIRA issue to track this if not already done, I
> wanted
> > some guidance from a committer or two to help guide me a bit take care of
> > at he administrative details and how to get this kicked off the ground.
> My
> > background includes working on apache reef codebase as well as dabbling
> in
> > apache mahout and apache flume as well as storm and various nosql/graph
> > databases. Looking forward to being part of the community.
> >
> >
> >
> > The rough plan of action in my mind so you know:
> >
> > 1) Review heron architecture and codebase in detail
> >
> > 2) Create an umbrella JIRA with the first series of tasks
> >
> > 3) Come up with a design doc
> >
> > 4) Build the interfaces
> >
> > 5) Add more dev tasks to JIRA
> >
> > 6) Drive the implementation
> >
> >
> > Thanks in advance for your help.
> >
> >
>
Re: Hello heron community
Posted by Sanjeev Kulkarni <sa...@gmail.com>.
I have a couple of questions
1. Do you plan on exposing both the low level(spouts/bolts) and the
streamlet api? Or are you preferring one over another?
2. My suggestion would be to start with the Streamlet interface. Primarily
because a) Most of scala ml libraries that I've seen tend to operate on the
streamlet kind of interface and b) It has a far smaller surface area(i.e.
number of interfaces) so might be easy to get something up quickly for
testing.
Thanks!
On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sx...@hotmail.com>
wrote:
> Thanks Sanjeev, I had an initial idea that I wanted to float on the list,
> I was thinking that as part of the initial scala port of the API I'd like
> to propose that we use this tool: http://javatoscala.com/
>
> Java to Scala converter<http://javatoscala.com/>
> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
> I propose that we use the above tool to do an initial conversion of all
> the Java functional and oo interfaces and then plug that into the initial
> design doc as well as the codebase to get things off the ground. I have
> already forked the heron repo and will be doing this off of a child branch
> based on my fork.
>
>
> How does that sound to folks, anyone has strong objections, of course I
> fully realize that I will still need to do a lot of work around refactoring
> the code even when using the converter so I'm prepared for that, the goal
> in using the converters is to save a bit of time so that manual
> intervention isnt necessarily needed in designing every interface.
>
> Thoughts.
>
> ________________________________
> From: Sanjeev Kulkarni <sa...@gmail.com>
> Sent: Sunday, October 29, 2017 10:18 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> Hi Saikat,
> Welcome to the Heron users group. Your plan of action sounds good to me. A
> native scala interface would be great for all those scala fans. Eagerly
> awaiting design doc.
> Thanks!
>
> On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
> wrote:
>
> > Hello Folks,
> >
> > I'm interested in taking and driving the following github component (this
> > means design/development/architecture and more).
> >
> >
> > https://github.com/twitter/heron/issues/668
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> >
> >
> > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 ...<
> > https://github.com/twitter/heron/issues/668>
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > heron - Heron is a realtime, distributed, fault-tolerant stream
> processing
> > engine from Twitter
> >
> >
> >
> >
> > I've joined the heron slack channel as recommended and am ready to start
> > by creating a master JIRA issue to track this if not already done, I
> wanted
> > some guidance from a committer or two to help guide me a bit take care of
> > at he administrative details and how to get this kicked off the ground.
> My
> > background includes working on apache reef codebase as well as dabbling
> in
> > apache mahout and apache flume as well as storm and various nosql/graph
> > databases. Looking forward to being part of the community.
> >
> >
> >
> > The rough plan of action in my mind so you know:
> >
> > 1) Review heron architecture and codebase in detail
> >
> > 2) Create an umbrella JIRA with the first series of tasks
> >
> > 3) Come up with a design doc
> >
> > 4) Build the interfaces
> >
> > 5) Add more dev tasks to JIRA
> >
> > 6) Drive the implementation
> >
> >
> > Thanks in advance for your help.
> >
> >
>
Re: Hello heron community
Posted by Saikat Kanjilal <sx...@hotmail.com>.
Thanks Sanjeev, I had an initial idea that I wanted to float on the list, I was thinking that as part of the initial scala port of the API I'd like to propose that we use this tool: http://javatoscala.com/
Java to Scala converter<http://javatoscala.com/>
javatoscala.com
How does it work? Java to Scala converter is created with Play framework and Scalagen library. I don't want you to see my code. No problem. Java to Scala converter ...
I propose that we use the above tool to do an initial conversion of all the Java functional and oo interfaces and then plug that into the initial design doc as well as the codebase to get things off the ground. I have already forked the heron repo and will be doing this off of a child branch based on my fork.
How does that sound to folks, anyone has strong objections, of course I fully realize that I will still need to do a lot of work around refactoring the code even when using the converter so I'm prepared for that, the goal in using the converters is to save a bit of time so that manual intervention isnt necessarily needed in designing every interface.
Thoughts.
________________________________
From: Sanjeev Kulkarni <sa...@gmail.com>
Sent: Sunday, October 29, 2017 10:18 PM
To: dev@heron.incubator.apache.org
Subject: Re: Hello heron community
Hi Saikat,
Welcome to the Heron users group. Your plan of action sounds good to me. A
native scala interface would be great for all those scala fans. Eagerly
awaiting design doc.
Thanks!
On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
wrote:
> Hello Folks,
>
> I'm interested in taking and driving the following github component (this
> means design/development/architecture and more).
>
>
> https://github.com/twitter/heron/issues/668
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
>
>
> [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 ...<
> https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>
Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.
> github.com
> heron - Heron is a realtime, distributed, fault-tolerant stream processing
> engine from Twitter
>
>
>
>
> I've joined the heron slack channel as recommended and am ready to start
> by creating a master JIRA issue to track this if not already done, I wanted
> some guidance from a committer or two to help guide me a bit take care of
> at he administrative details and how to get this kicked off the ground. My
> background includes working on apache reef codebase as well as dabbling in
> apache mahout and apache flume as well as storm and various nosql/graph
> databases. Looking forward to being part of the community.
>
>
>
> The rough plan of action in my mind so you know:
>
> 1) Review heron architecture and codebase in detail
>
> 2) Create an umbrella JIRA with the first series of tasks
>
> 3) Come up with a design doc
>
> 4) Build the interfaces
>
> 5) Add more dev tasks to JIRA
>
> 6) Drive the implementation
>
>
> Thanks in advance for your help.
>
>
Re: Hello heron community
Posted by Sanjeev Kulkarni <sa...@gmail.com>.
Hi Saikat,
Welcome to the Heron users group. Your plan of action sounds good to me. A
native scala interface would be great for all those scala fans. Eagerly
awaiting design doc.
Thanks!
On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sx...@hotmail.com>
wrote:
> Hello Folks,
>
> I'm interested in taking and driving the following github component (this
> means design/development/architecture and more).
>
>
> https://github.com/twitter/heron/issues/668
>
>
> [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 ...<
> https://github.com/twitter/heron/issues/668>
> github.com
> heron - Heron is a realtime, distributed, fault-tolerant stream processing
> engine from Twitter
>
>
>
>
> I've joined the heron slack channel as recommended and am ready to start
> by creating a master JIRA issue to track this if not already done, I wanted
> some guidance from a committer or two to help guide me a bit take care of
> at he administrative details and how to get this kicked off the ground. My
> background includes working on apache reef codebase as well as dabbling in
> apache mahout and apache flume as well as storm and various nosql/graph
> databases. Looking forward to being part of the community.
>
>
>
> The rough plan of action in my mind so you know:
>
> 1) Review heron architecture and codebase in detail
>
> 2) Create an umbrella JIRA with the first series of tasks
>
> 3) Come up with a design doc
>
> 4) Build the interfaces
>
> 5) Add more dev tasks to JIRA
>
> 6) Drive the implementation
>
>
> Thanks in advance for your help.
>
>