You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Tushar Gosavi <tu...@datatorrent.com> on 2015/09/28 16:29:39 UTC
Writing Streaming application in Scala.
Hi All,
I am trying to see how we can write a Streaming Application using Scala
language,
as Scala targets JVM and has good interoperability with Java.
I am able to port simple pi demo in Scala along with the operators. Please
take
a look at PR#40 (https://github.com/apache/incubator-apex-malhar/pull/40)
for this work, and provide feedback.
Created following operators in Scala
- RandomItemGenerator : Generate random points in 2d space.
- PiCalculator : Calculates value of pi
- ConsoleOutOperator : Write value of tuple on stdout, I could not use
the Java operator available in Malhar, because scala does more strong type
checking. (I need to look into this more)
And use these operators to write a simple streaming application.
The following functionality is working
- Scala Operators recovers correctly from the crash.
- Scala supports generating setters/getters using @BeanProperty annotation,
and these
properties become accessible through config file and we can set them
dynamically through dtcli.
- get-operator-properties is working.
Thanks,
-Tushar.
Re: Writing Streaming application in Scala.
Posted by Siyuan Hua <si...@datatorrent.com>.
Hi Tushar,
This is the jira https://malhar.atlassian.net/browse/APEX-175
Siyuan
On Tue, Sep 29, 2015 at 9:02 PM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> I can take up this work. Siyun let me know if you have already opened a
> Jira for this, else I will open one today.
>
> - Tushar.
>
> On Wed, Sep 30, 2015 at 2:08 AM, Chetan Narsude <ch...@datatorrent.com>
> wrote:
>
> > +1
> >
> > The current tutorial just tells the Scala Apex developer to walk. Scala
> > tuned API can make them run.
> >
> > Siyuan, why don't you open a JIRA issue to track it so someone interested
> > can take it?
> >
> > --
> > Chetan
> >
> > On Tue, Sep 29, 2015 at 1:27 PM, Siyuan Hua <si...@datatorrent.com>
> > wrote:
> >
> > > This is a good example. But the code here is still in Java style.
> > > I wish we could have another abstraction of scala api in the roadmap
> that
> > > could leverage the functional paradigm to write code in a concise way:
> > > for example:
> > > A simple statement like this
> > > stream.map(filter(tuple) -> {}, 5).reduce(aggregate(tuple, tuple)->{},
> 1)
> > > could spin up 5 filter operators and 1 aggregation operator
> > >
> > > Siyuan
> > >
> > >
> > > On Tue, Sep 29, 2015 at 7:40 AM, Munagala Ramanath <
> ram@datatorrent.com>
> > > wrote:
> > >
> > > > Languages like Scala, Clojure and dialects like Jython and JRuby are
> > > easier
> > > > to
> > > > support since they are JVM based and can interoperate with Java.
> > > >
> > > > Non JVM lanugages are a lot more work to support and also present
> > > > significant
> > > > performance issues because converting objects across the JVM boundary
> > is
> > > > often a heavy duty process.
> > > >
> > > > Ram
> > > >
> > > > On Mon, Sep 28, 2015 at 11:36 PM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com
> > > > > wrote:
> > > >
> > > > > This is great. Would it also make sense to have similar using
> native
> > > > > language like C++ using JNI bindings.
> > > > >
> > > > > To my knowledge, complex enterprise based applications prefer
> > language
> > > > > closer to native for high performance and better memory
> utilization.
> > > > > If we could get writing a streaming application in C++ along with
> > power
> > > > of
> > > > > Apex, that would add a lot of value in my opinion.
> > > > >
> > > > > Any thoughts?
> > > > >
> > > > >
> > > > > ~ Chinmay.
> > > > >
> > > > > On Mon, Sep 28, 2015 at 9:06 PM, Munagala Ramanath <
> > > ram@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Tushar,
> > > > > >
> > > > > > This is a great project to add to dtBackyard at:
> > > > > >
> > > >
> > https://sites.google.com/a/datatorrent.com/dtbackyard/project-ownership
> > > > > >
> > > > > > Ram
> > > > > >
> > > > > > On Mon, Sep 28, 2015 at 7:29 AM, Tushar Gosavi <
> > > tushar@datatorrent.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I am trying to see how we can write a Streaming Application
> using
> > > > Scala
> > > > > > > language,
> > > > > > > as Scala targets JVM and has good interoperability with Java.
> > > > > > >
> > > > > > > I am able to port simple pi demo in Scala along with the
> > operators.
> > > > > > Please
> > > > > > > take
> > > > > > > a look at PR#40 (
> > > > > https://github.com/apache/incubator-apex-malhar/pull/40
> > > > > > )
> > > > > > > for this work, and provide feedback.
> > > > > > >
> > > > > > > Created following operators in Scala
> > > > > > > - RandomItemGenerator : Generate random points in 2d space.
> > > > > > > - PiCalculator : Calculates value of pi
> > > > > > > - ConsoleOutOperator : Write value of tuple on stdout, I
> could
> > > not
> > > > > use
> > > > > > > the Java operator available in Malhar, because scala does more
> > > strong
> > > > > > type
> > > > > > > checking. (I need to look into this more)
> > > > > > >
> > > > > > > And use these operators to write a simple streaming
> application.
> > > > > > >
> > > > > > > The following functionality is working
> > > > > > > - Scala Operators recovers correctly from the crash.
> > > > > > > - Scala supports generating setters/getters using @BeanProperty
> > > > > > annotation,
> > > > > > > and these
> > > > > > > properties become accessible through config file and we can
> set
> > > > them
> > > > > > > dynamically through dtcli.
> > > > > > > - get-operator-properties is working.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -Tushar.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Writing Streaming application in Scala.
Posted by Tushar Gosavi <tu...@datatorrent.com>.
I can take up this work. Siyun let me know if you have already opened a
Jira for this, else I will open one today.
- Tushar.
On Wed, Sep 30, 2015 at 2:08 AM, Chetan Narsude <ch...@datatorrent.com>
wrote:
> +1
>
> The current tutorial just tells the Scala Apex developer to walk. Scala
> tuned API can make them run.
>
> Siyuan, why don't you open a JIRA issue to track it so someone interested
> can take it?
>
> --
> Chetan
>
> On Tue, Sep 29, 2015 at 1:27 PM, Siyuan Hua <si...@datatorrent.com>
> wrote:
>
> > This is a good example. But the code here is still in Java style.
> > I wish we could have another abstraction of scala api in the roadmap that
> > could leverage the functional paradigm to write code in a concise way:
> > for example:
> > A simple statement like this
> > stream.map(filter(tuple) -> {}, 5).reduce(aggregate(tuple, tuple)->{}, 1)
> > could spin up 5 filter operators and 1 aggregation operator
> >
> > Siyuan
> >
> >
> > On Tue, Sep 29, 2015 at 7:40 AM, Munagala Ramanath <ra...@datatorrent.com>
> > wrote:
> >
> > > Languages like Scala, Clojure and dialects like Jython and JRuby are
> > easier
> > > to
> > > support since they are JVM based and can interoperate with Java.
> > >
> > > Non JVM lanugages are a lot more work to support and also present
> > > significant
> > > performance issues because converting objects across the JVM boundary
> is
> > > often a heavy duty process.
> > >
> > > Ram
> > >
> > > On Mon, Sep 28, 2015 at 11:36 PM, Chinmay Kolhatkar <
> > > chinmay@datatorrent.com
> > > > wrote:
> > >
> > > > This is great. Would it also make sense to have similar using native
> > > > language like C++ using JNI bindings.
> > > >
> > > > To my knowledge, complex enterprise based applications prefer
> language
> > > > closer to native for high performance and better memory utilization.
> > > > If we could get writing a streaming application in C++ along with
> power
> > > of
> > > > Apex, that would add a lot of value in my opinion.
> > > >
> > > > Any thoughts?
> > > >
> > > >
> > > > ~ Chinmay.
> > > >
> > > > On Mon, Sep 28, 2015 at 9:06 PM, Munagala Ramanath <
> > ram@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Hi Tushar,
> > > > >
> > > > > This is a great project to add to dtBackyard at:
> > > > >
> > >
> https://sites.google.com/a/datatorrent.com/dtbackyard/project-ownership
> > > > >
> > > > > Ram
> > > > >
> > > > > On Mon, Sep 28, 2015 at 7:29 AM, Tushar Gosavi <
> > tushar@datatorrent.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I am trying to see how we can write a Streaming Application using
> > > Scala
> > > > > > language,
> > > > > > as Scala targets JVM and has good interoperability with Java.
> > > > > >
> > > > > > I am able to port simple pi demo in Scala along with the
> operators.
> > > > > Please
> > > > > > take
> > > > > > a look at PR#40 (
> > > > https://github.com/apache/incubator-apex-malhar/pull/40
> > > > > )
> > > > > > for this work, and provide feedback.
> > > > > >
> > > > > > Created following operators in Scala
> > > > > > - RandomItemGenerator : Generate random points in 2d space.
> > > > > > - PiCalculator : Calculates value of pi
> > > > > > - ConsoleOutOperator : Write value of tuple on stdout, I could
> > not
> > > > use
> > > > > > the Java operator available in Malhar, because scala does more
> > strong
> > > > > type
> > > > > > checking. (I need to look into this more)
> > > > > >
> > > > > > And use these operators to write a simple streaming application.
> > > > > >
> > > > > > The following functionality is working
> > > > > > - Scala Operators recovers correctly from the crash.
> > > > > > - Scala supports generating setters/getters using @BeanProperty
> > > > > annotation,
> > > > > > and these
> > > > > > properties become accessible through config file and we can set
> > > them
> > > > > > dynamically through dtcli.
> > > > > > - get-operator-properties is working.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > -Tushar.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Writing Streaming application in Scala.
Posted by Chetan Narsude <ch...@datatorrent.com>.
+1
The current tutorial just tells the Scala Apex developer to walk. Scala
tuned API can make them run.
Siyuan, why don't you open a JIRA issue to track it so someone interested
can take it?
--
Chetan
On Tue, Sep 29, 2015 at 1:27 PM, Siyuan Hua <si...@datatorrent.com> wrote:
> This is a good example. But the code here is still in Java style.
> I wish we could have another abstraction of scala api in the roadmap that
> could leverage the functional paradigm to write code in a concise way:
> for example:
> A simple statement like this
> stream.map(filter(tuple) -> {}, 5).reduce(aggregate(tuple, tuple)->{}, 1)
> could spin up 5 filter operators and 1 aggregation operator
>
> Siyuan
>
>
> On Tue, Sep 29, 2015 at 7:40 AM, Munagala Ramanath <ra...@datatorrent.com>
> wrote:
>
> > Languages like Scala, Clojure and dialects like Jython and JRuby are
> easier
> > to
> > support since they are JVM based and can interoperate with Java.
> >
> > Non JVM lanugages are a lot more work to support and also present
> > significant
> > performance issues because converting objects across the JVM boundary is
> > often a heavy duty process.
> >
> > Ram
> >
> > On Mon, Sep 28, 2015 at 11:36 PM, Chinmay Kolhatkar <
> > chinmay@datatorrent.com
> > > wrote:
> >
> > > This is great. Would it also make sense to have similar using native
> > > language like C++ using JNI bindings.
> > >
> > > To my knowledge, complex enterprise based applications prefer language
> > > closer to native for high performance and better memory utilization.
> > > If we could get writing a streaming application in C++ along with power
> > of
> > > Apex, that would add a lot of value in my opinion.
> > >
> > > Any thoughts?
> > >
> > >
> > > ~ Chinmay.
> > >
> > > On Mon, Sep 28, 2015 at 9:06 PM, Munagala Ramanath <
> ram@datatorrent.com>
> > > wrote:
> > >
> > > > Hi Tushar,
> > > >
> > > > This is a great project to add to dtBackyard at:
> > > >
> > https://sites.google.com/a/datatorrent.com/dtbackyard/project-ownership
> > > >
> > > > Ram
> > > >
> > > > On Mon, Sep 28, 2015 at 7:29 AM, Tushar Gosavi <
> tushar@datatorrent.com
> > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I am trying to see how we can write a Streaming Application using
> > Scala
> > > > > language,
> > > > > as Scala targets JVM and has good interoperability with Java.
> > > > >
> > > > > I am able to port simple pi demo in Scala along with the operators.
> > > > Please
> > > > > take
> > > > > a look at PR#40 (
> > > https://github.com/apache/incubator-apex-malhar/pull/40
> > > > )
> > > > > for this work, and provide feedback.
> > > > >
> > > > > Created following operators in Scala
> > > > > - RandomItemGenerator : Generate random points in 2d space.
> > > > > - PiCalculator : Calculates value of pi
> > > > > - ConsoleOutOperator : Write value of tuple on stdout, I could
> not
> > > use
> > > > > the Java operator available in Malhar, because scala does more
> strong
> > > > type
> > > > > checking. (I need to look into this more)
> > > > >
> > > > > And use these operators to write a simple streaming application.
> > > > >
> > > > > The following functionality is working
> > > > > - Scala Operators recovers correctly from the crash.
> > > > > - Scala supports generating setters/getters using @BeanProperty
> > > > annotation,
> > > > > and these
> > > > > properties become accessible through config file and we can set
> > them
> > > > > dynamically through dtcli.
> > > > > - get-operator-properties is working.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > -Tushar.
> > > > >
> > > >
> > >
> >
>
Re: Writing Streaming application in Scala.
Posted by Siyuan Hua <si...@datatorrent.com>.
This is a good example. But the code here is still in Java style.
I wish we could have another abstraction of scala api in the roadmap that
could leverage the functional paradigm to write code in a concise way:
for example:
A simple statement like this
stream.map(filter(tuple) -> {}, 5).reduce(aggregate(tuple, tuple)->{}, 1)
could spin up 5 filter operators and 1 aggregation operator
Siyuan
On Tue, Sep 29, 2015 at 7:40 AM, Munagala Ramanath <ra...@datatorrent.com>
wrote:
> Languages like Scala, Clojure and dialects like Jython and JRuby are easier
> to
> support since they are JVM based and can interoperate with Java.
>
> Non JVM lanugages are a lot more work to support and also present
> significant
> performance issues because converting objects across the JVM boundary is
> often a heavy duty process.
>
> Ram
>
> On Mon, Sep 28, 2015 at 11:36 PM, Chinmay Kolhatkar <
> chinmay@datatorrent.com
> > wrote:
>
> > This is great. Would it also make sense to have similar using native
> > language like C++ using JNI bindings.
> >
> > To my knowledge, complex enterprise based applications prefer language
> > closer to native for high performance and better memory utilization.
> > If we could get writing a streaming application in C++ along with power
> of
> > Apex, that would add a lot of value in my opinion.
> >
> > Any thoughts?
> >
> >
> > ~ Chinmay.
> >
> > On Mon, Sep 28, 2015 at 9:06 PM, Munagala Ramanath <ra...@datatorrent.com>
> > wrote:
> >
> > > Hi Tushar,
> > >
> > > This is a great project to add to dtBackyard at:
> > >
> https://sites.google.com/a/datatorrent.com/dtbackyard/project-ownership
> > >
> > > Ram
> > >
> > > On Mon, Sep 28, 2015 at 7:29 AM, Tushar Gosavi <tushar@datatorrent.com
> >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am trying to see how we can write a Streaming Application using
> Scala
> > > > language,
> > > > as Scala targets JVM and has good interoperability with Java.
> > > >
> > > > I am able to port simple pi demo in Scala along with the operators.
> > > Please
> > > > take
> > > > a look at PR#40 (
> > https://github.com/apache/incubator-apex-malhar/pull/40
> > > )
> > > > for this work, and provide feedback.
> > > >
> > > > Created following operators in Scala
> > > > - RandomItemGenerator : Generate random points in 2d space.
> > > > - PiCalculator : Calculates value of pi
> > > > - ConsoleOutOperator : Write value of tuple on stdout, I could not
> > use
> > > > the Java operator available in Malhar, because scala does more strong
> > > type
> > > > checking. (I need to look into this more)
> > > >
> > > > And use these operators to write a simple streaming application.
> > > >
> > > > The following functionality is working
> > > > - Scala Operators recovers correctly from the crash.
> > > > - Scala supports generating setters/getters using @BeanProperty
> > > annotation,
> > > > and these
> > > > properties become accessible through config file and we can set
> them
> > > > dynamically through dtcli.
> > > > - get-operator-properties is working.
> > > >
> > > >
> > > > Thanks,
> > > > -Tushar.
> > > >
> > >
> >
>
Re: Writing Streaming application in Scala.
Posted by Munagala Ramanath <ra...@datatorrent.com>.
Languages like Scala, Clojure and dialects like Jython and JRuby are easier
to
support since they are JVM based and can interoperate with Java.
Non JVM lanugages are a lot more work to support and also present
significant
performance issues because converting objects across the JVM boundary is
often a heavy duty process.
Ram
On Mon, Sep 28, 2015 at 11:36 PM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:
> This is great. Would it also make sense to have similar using native
> language like C++ using JNI bindings.
>
> To my knowledge, complex enterprise based applications prefer language
> closer to native for high performance and better memory utilization.
> If we could get writing a streaming application in C++ along with power of
> Apex, that would add a lot of value in my opinion.
>
> Any thoughts?
>
>
> ~ Chinmay.
>
> On Mon, Sep 28, 2015 at 9:06 PM, Munagala Ramanath <ra...@datatorrent.com>
> wrote:
>
> > Hi Tushar,
> >
> > This is a great project to add to dtBackyard at:
> > https://sites.google.com/a/datatorrent.com/dtbackyard/project-ownership
> >
> > Ram
> >
> > On Mon, Sep 28, 2015 at 7:29 AM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I am trying to see how we can write a Streaming Application using Scala
> > > language,
> > > as Scala targets JVM and has good interoperability with Java.
> > >
> > > I am able to port simple pi demo in Scala along with the operators.
> > Please
> > > take
> > > a look at PR#40 (
> https://github.com/apache/incubator-apex-malhar/pull/40
> > )
> > > for this work, and provide feedback.
> > >
> > > Created following operators in Scala
> > > - RandomItemGenerator : Generate random points in 2d space.
> > > - PiCalculator : Calculates value of pi
> > > - ConsoleOutOperator : Write value of tuple on stdout, I could not
> use
> > > the Java operator available in Malhar, because scala does more strong
> > type
> > > checking. (I need to look into this more)
> > >
> > > And use these operators to write a simple streaming application.
> > >
> > > The following functionality is working
> > > - Scala Operators recovers correctly from the crash.
> > > - Scala supports generating setters/getters using @BeanProperty
> > annotation,
> > > and these
> > > properties become accessible through config file and we can set them
> > > dynamically through dtcli.
> > > - get-operator-properties is working.
> > >
> > >
> > > Thanks,
> > > -Tushar.
> > >
> >
>
Re: Writing Streaming application in Scala.
Posted by Chinmay Kolhatkar <ch...@datatorrent.com>.
This is great. Would it also make sense to have similar using native
language like C++ using JNI bindings.
To my knowledge, complex enterprise based applications prefer language
closer to native for high performance and better memory utilization.
If we could get writing a streaming application in C++ along with power of
Apex, that would add a lot of value in my opinion.
Any thoughts?
~ Chinmay.
On Mon, Sep 28, 2015 at 9:06 PM, Munagala Ramanath <ra...@datatorrent.com>
wrote:
> Hi Tushar,
>
> This is a great project to add to dtBackyard at:
> https://sites.google.com/a/datatorrent.com/dtbackyard/project-ownership
>
> Ram
>
> On Mon, Sep 28, 2015 at 7:29 AM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > Hi All,
> >
> > I am trying to see how we can write a Streaming Application using Scala
> > language,
> > as Scala targets JVM and has good interoperability with Java.
> >
> > I am able to port simple pi demo in Scala along with the operators.
> Please
> > take
> > a look at PR#40 (https://github.com/apache/incubator-apex-malhar/pull/40
> )
> > for this work, and provide feedback.
> >
> > Created following operators in Scala
> > - RandomItemGenerator : Generate random points in 2d space.
> > - PiCalculator : Calculates value of pi
> > - ConsoleOutOperator : Write value of tuple on stdout, I could not use
> > the Java operator available in Malhar, because scala does more strong
> type
> > checking. (I need to look into this more)
> >
> > And use these operators to write a simple streaming application.
> >
> > The following functionality is working
> > - Scala Operators recovers correctly from the crash.
> > - Scala supports generating setters/getters using @BeanProperty
> annotation,
> > and these
> > properties become accessible through config file and we can set them
> > dynamically through dtcli.
> > - get-operator-properties is working.
> >
> >
> > Thanks,
> > -Tushar.
> >
>
Re: Writing Streaming application in Scala.
Posted by Munagala Ramanath <ra...@datatorrent.com>.
Hi Tushar,
This is a great project to add to dtBackyard at:
https://sites.google.com/a/datatorrent.com/dtbackyard/project-ownership
Ram
On Mon, Sep 28, 2015 at 7:29 AM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> Hi All,
>
> I am trying to see how we can write a Streaming Application using Scala
> language,
> as Scala targets JVM and has good interoperability with Java.
>
> I am able to port simple pi demo in Scala along with the operators. Please
> take
> a look at PR#40 (https://github.com/apache/incubator-apex-malhar/pull/40)
> for this work, and provide feedback.
>
> Created following operators in Scala
> - RandomItemGenerator : Generate random points in 2d space.
> - PiCalculator : Calculates value of pi
> - ConsoleOutOperator : Write value of tuple on stdout, I could not use
> the Java operator available in Malhar, because scala does more strong type
> checking. (I need to look into this more)
>
> And use these operators to write a simple streaming application.
>
> The following functionality is working
> - Scala Operators recovers correctly from the crash.
> - Scala supports generating setters/getters using @BeanProperty annotation,
> and these
> properties become accessible through config file and we can set them
> dynamically through dtcli.
> - get-operator-properties is working.
>
>
> Thanks,
> -Tushar.
>