You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by nunarob <ro...@nunahealth.com> on 2014/07/31 02:20:52 UTC
Deploying spark applications from within Eclipse?
Hi all,
I'm a new Spark user and I'm interested in deploying java-based Spark
applications directly from Eclipse. Specifically, I'd like to write a
Spark/java application, have it directly submitted to a spark cluster, and
then see the output of the spark execution directly in the eclipse console.
Following the quick start guide, I'm able to package up a spark application
using maven from within eclipse and then able to manually submit the jar
using the spark-submit script. Is there anyway to submit the packaged jar to
the spark cluster directly from within eclipse, and to see the output in the
eclipse console?
Apologies if this is documented somewhere - still learning about the entire
spark ecosystem.
Thanks!
-Rob
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Deploying spark applications from within Eclipse?
Posted by nunarob <ro...@nunahealth.com>.
Thanks much for your help Tobias.
I am trying what you have suggested but am receiving error messages which I
think are generally related to incorrect spark configuration. Will try and
fix those and let you know how it goes.
Cheers,
-Rob
On Wed, Jul 30, 2014 at 6:01 PM, Tobias Pfeiffer [via Apache Spark User
List] <ml...@n3.nabble.com> wrote:
> Hi,
>
> if your Spark driver runs in Eclipse, you should be able to get what you
> want. That is, create a main() function that creates a SparkContext with
> the correct master URL of your cluster and execute it with your standard
> Eclipse method. For example, I am using Mesos and the following works fine:
>
> val conf = new SparkConf()
> .setMaster("zk://HOSTNAME:2181/mesos")
> .setAppName("My Application")
> .setJars("/path/to/my-application-assembly-1.0.jar" :: Nil)
> .set("spark.executor.uri",
> "/path/to/spark-1.0.0-bin-hadoop2.tgz")
> .set("spark.mesos.coarse", "true")
> val sc = new SparkContext(conf)
>
> However, if you change something in the cluster node processing code, you
> will have to redeploy the my-application-assembly-1.0.jar file.
>
> Tobias
>
>
> On Thu, Jul 31, 2014 at 9:20 AM, nunarob <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=11007&i=0>> wrote:
>
>> Hi all,
>>
>> I'm a new Spark user and I'm interested in deploying java-based Spark
>> applications directly from Eclipse. Specifically, I'd like to write a
>> Spark/java application, have it directly submitted to a spark cluster, and
>> then see the output of the spark execution directly in the eclipse
>> console.
>>
>> Following the quick start guide, I'm able to package up a spark
>> application
>> using maven from within eclipse and then able to manually submit the jar
>> using the spark-submit script. Is there anyway to submit the packaged jar
>> to
>> the spark cluster directly from within eclipse, and to see the output in
>> the
>> eclipse console?
>>
>> Apologies if this is documented somewhere - still learning about the
>> entire
>> spark ecosystem.
>>
>> Thanks!
>>
>> -Rob
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001p11007.html
> To unsubscribe from Deploying spark applications from within Eclipse?, click
> here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=11001&code=cm9iQG51bmFoZWFsdGguY29tfDExMDAxfC00ODcwNDM5MTI=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001p11190.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Deploying spark applications from within Eclipse?
Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Hi,
if your Spark driver runs in Eclipse, you should be able to get what you
want. That is, create a main() function that creates a SparkContext with
the correct master URL of your cluster and execute it with your standard
Eclipse method. For example, I am using Mesos and the following works fine:
val conf = new SparkConf()
.setMaster("zk://HOSTNAME:2181/mesos")
.setAppName("My Application")
.setJars("/path/to/my-application-assembly-1.0.jar" :: Nil)
.set("spark.executor.uri", "/path/to/spark-1.0.0-bin-hadoop2.tgz")
.set("spark.mesos.coarse", "true")
val sc = new SparkContext(conf)
However, if you change something in the cluster node processing code, you
will have to redeploy the my-application-assembly-1.0.jar file.
Tobias
On Thu, Jul 31, 2014 at 9:20 AM, nunarob <ro...@nunahealth.com> wrote:
> Hi all,
>
> I'm a new Spark user and I'm interested in deploying java-based Spark
> applications directly from Eclipse. Specifically, I'd like to write a
> Spark/java application, have it directly submitted to a spark cluster, and
> then see the output of the spark execution directly in the eclipse console.
>
> Following the quick start guide, I'm able to package up a spark application
> using maven from within eclipse and then able to manually submit the jar
> using the spark-submit script. Is there anyway to submit the packaged jar
> to
> the spark cluster directly from within eclipse, and to see the output in
> the
> eclipse console?
>
> Apologies if this is documented somewhere - still learning about the entire
> spark ecosystem.
>
> Thanks!
>
> -Rob
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>