You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Zwits <Da...@ortec-finance.com> on 2015/07/02 10:33:18 UTC
.NET on Apache Spark?
I'm currently looking into a way to run a program/code (DAG) written in .NET
on a cluster using Spark. However I ran into problems concerning the coding
language, Spark has no .NET API.
I tried looking into IronPython because Spark does have a Python API, but i
couldn't find a way to use this.
Is there a way to implement a DAG of jobs on a cluster using .NET
programming language?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NET-on-Apache-Spark-tp23578.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: .NET on Apache Spark?
Posted by Silvio Fiorito <si...@granturing.com>.
Joe Duffy, director of engineering on Microsoft's compiler team made a comment about investigating F# type providers for Spark. https://twitter.com/xjoeduffyx/status/614076012372955136
From: Ashic Mahtab<ma...@live.com>
Sent: ?Sunday?, ?July? ?5?, ?2015 ?1?:?29? ?PM
To: Ruslan Dautkhanov<ma...@gmail.com>, pedro<ma...@gmail.com>
Cc: user@spark.apache.org<ma...@spark.apache.org>
Unfortunately, afaik that project is long dead.
It'd be an interesting project to create an intermediary protocol, perhaps using something that nearly everything these days understand (unfortunately [!] that might be JavaScript). For example, instead of pickling language constructs, it might be interesting to translate rdd operations to some json structure, and have a single thing server side processing the "instructions".
There's also mbrace (http://www.m-brace.net/)... mbrace-spark integration would be quite interesting indeed. Though the difference in approach might be quite a challenge.
Another approach could be using IKVM to host the JVM, much like how pyspark executes.
Microsoft research published some very early work in OneNet: http://research.microsoft.com/en-us/um/people/jinl/redesign/research/onenet_executive_summary.pdf - their careers page seems to be recruiting for the project.
Again, these are all future things, most of which would need to be community driven. If you need something right now, then there really isn't good integration between spark and .NET. However, given your requirements, mbrace might be something that you might find useful.
-Ashic.
________________________________
Date: Sun, 5 Jul 2015 11:05:30 -0600
Subject: Re: .NET on Apache Spark?
From: dautkhanov@gmail.com
To: ski.rodriguez@gmail.com
CC: user@spark.apache.org
Scala used to run on .NET
http://www.scala-lang.org/old/node/10299
--
Ruslan Dautkhanov
On Thu, Jul 2, 2015 at 1:26 PM, pedro <sk...@gmail.com>> wrote:
You might try using .pipe() and installing your .NET program as a binary
across the cluster (or using addFile). Its not ideal to pipe things in/out
along with the overhead, but it would work.
I don't know much about IronPython, but perhaps changing the default python
by changing your path might work?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NET-on-Apache-Spark-tp23578p23594.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<ma...@spark.apache.org>
RE: .NET on Apache Spark?
Posted by Ashic Mahtab <as...@live.com>.
Unfortunately, afaik that project is long dead.
It'd be an interesting project to create an intermediary protocol, perhaps using something that nearly everything these days understand (unfortunately [!] that might be JavaScript). For example, instead of pickling language constructs, it might be interesting to translate rdd operations to some json structure, and have a single thing server side processing the "instructions".
There's also mbrace (http://www.m-brace.net/)... mbrace-spark integration would be quite interesting indeed. Though the difference in approach might be quite a challenge.
Another approach could be using IKVM to host the JVM, much like how pyspark executes.
Microsoft research published some very early work in OneNet: http://research.microsoft.com/en-us/um/people/jinl/redesign/research/onenet_executive_summary.pdf - their careers page seems to be recruiting for the project.
Again, these are all future things, most of which would need to be community driven. If you need something right now, then there really isn't good integration between spark and .NET. However, given your requirements, mbrace might be something that you might find useful.
-Ashic.
Date: Sun, 5 Jul 2015 11:05:30 -0600
Subject: Re: .NET on Apache Spark?
From: dautkhanov@gmail.com
To: ski.rodriguez@gmail.com
CC: user@spark.apache.org
Scala used to run on .NEThttp://www.scala-lang.org/old/node/10299
--
Ruslan Dautkhanov
On Thu, Jul 2, 2015 at 1:26 PM, pedro <sk...@gmail.com> wrote:
You might try using .pipe() and installing your .NET program as a binary
across the cluster (or using addFile). Its not ideal to pipe things in/out
along with the overhead, but it would work.
I don't know much about IronPython, but perhaps changing the default python
by changing your path might work?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NET-on-Apache-Spark-tp23578p23594.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: .NET on Apache Spark?
Posted by Ruslan Dautkhanov <da...@gmail.com>.
Scala used to run on .NET
http://www.scala-lang.org/old/node/10299
--
Ruslan Dautkhanov
On Thu, Jul 2, 2015 at 1:26 PM, pedro <sk...@gmail.com> wrote:
> You might try using .pipe() and installing your .NET program as a binary
> across the cluster (or using addFile). Its not ideal to pipe things in/out
> along with the overhead, but it would work.
>
> I don't know much about IronPython, but perhaps changing the default python
> by changing your path might work?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NET-on-Apache-Spark-tp23578p23594.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: .NET on Apache Spark?
Posted by pedro <sk...@gmail.com>.
You might try using .pipe() and installing your .NET program as a binary
across the cluster (or using addFile). Its not ideal to pipe things in/out
along with the overhead, but it would work.
I don't know much about IronPython, but perhaps changing the default python
by changing your path might work?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NET-on-Apache-Spark-tp23578p23594.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: .NET on Apache Spark?
Posted by Silvio Fiorito <si...@granturing.com>.
Since Spark runs on the JVM, no there isn't support for .Net.
You should take a look at Dryad and Naiad instead.
https://github.com/MicrosoftResearch/
________________________________
From: Zwits<ma...@ortec-finance.com>
Sent: 7/2/2015 4:33 AM
To: user@spark.apache.org<ma...@spark.apache.org>
Subject: .NET on Apache Spark?
I'm currently looking into a way to run a program/code (DAG) written in .NET
on a cluster using Spark. However I ran into problems concerning the coding
language, Spark has no .NET API.
I tried looking into IronPython because Spark does have a Python API, but i
couldn't find a way to use this.
Is there a way to implement a DAG of jobs on a cluster using .NET
programming language?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NET-on-Apache-Spark-tp23578.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: .NET on Apache Spark?
Posted by Jörn Franke <jo...@gmail.com>.
Ironpython shares with python only the syntax - at best. It is a scripting
language within the .NET framework. Many applications have this for
scripting the application itself. This won't work for you. You can use
pipes or write your spark jobs in java/scala/r and submit them via your
.NET framework. Alternatively, you create java spark jobs as skeleton and
you call your .NET libraries via jni. Depends a little bit on your use
case. From an architecture point of view you should be careful about memory
management in these constellations...
Le jeu. 2 juil. 2015 à 10:33, Zwits <Da...@ortec-finance.com> a
écrit :
> I'm currently looking into a way to run a program/code (DAG) written in
> .NET
> on a cluster using Spark. However I ran into problems concerning the coding
> language, Spark has no .NET API.
> I tried looking into IronPython because Spark does have a Python API, but i
> couldn't find a way to use this.
>
> Is there a way to implement a DAG of jobs on a cluster using .NET
> programming language?
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NET-on-Apache-Spark-tp23578.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: .NET on Apache Spark?
Posted by Daniel Darabos <da...@lynxanalytics.com>.
Indeed Spark does not have .NET bindings.
On Thu, Jul 2, 2015 at 10:33 AM, Zwits <Da...@ortec-finance.com>
wrote:
> I'm currently looking into a way to run a program/code (DAG) written in
> .NET
> on a cluster using Spark. However I ran into problems concerning the coding
> language, Spark has no .NET API.
> I tried looking into IronPython because Spark does have a Python API, but i
> couldn't find a way to use this.
>
> Is there a way to implement a DAG of jobs on a cluster using .NET
> programming language?
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NET-on-Apache-Spark-tp23578.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>