You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by rrussell25 <rr...@gmail.com> on 2015/03/16 20:57:19 UTC
Re: What is best way to run spark job in "yarn-cluster" mode from
java program(servlet container) and NOT using spark-submit command.
Hi, were you ever able to determine a satisfactory approach for this problem?
I have a similar situation and would prefer to execute the job directly from
java code within my jms listener and/or servlet container.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/What-is-best-way-to-run-spark-job-in-yarn-cluster-mode-from-java-program-servlet-container-and-NOT-u-tp21817p22086.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: What is best way to run spark job in "yarn-cluster" mode from java program(servlet container) and NOT using spark-submit command.
Posted by Noorul Islam K M <no...@noorul.com>.
Sandy Ryza <sa...@cloudera.com> writes:
> Creating a SparkContext and setting master as yarn-cluster unfortunately
> will not work.
>
> SPARK-4924 added APIs for doing this in Spark, but won't be included until
> 1.4.
>
> -Sandy
>
Did you look into something like [1]? With that you can make rest API
call from your java code.
Thanks and Regards
Noorul
[1] https://github.com/spark-jobserver/spark-jobserver?
> On Tue, Mar 17, 2015 at 3:19 AM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Create SparkContext set master as yarn-cluster then run it as a standalone
>> program?
>>
>> Thanks
>> Best Regards
>>
>> On Tue, Mar 17, 2015 at 1:27 AM, rrussell25 <rr...@gmail.com> wrote:
>>
>>> Hi, were you ever able to determine a satisfactory approach for this
>>> problem?
>>> I have a similar situation and would prefer to execute the job directly
>>> from
>>> java code within my jms listener and/or servlet container.
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/What-is-best-way-to-run-spark-job-in-yarn-cluster-mode-from-java-program-servlet-container-and-NOT-u-tp21817p22086.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: What is best way to run spark job in "yarn-cluster" mode from
java program(servlet container) and NOT using spark-submit command.
Posted by Sandy Ryza <sa...@cloudera.com>.
Creating a SparkContext and setting master as yarn-cluster unfortunately
will not work.
SPARK-4924 added APIs for doing this in Spark, but won't be included until
1.4.
-Sandy
On Tue, Mar 17, 2015 at 3:19 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Create SparkContext set master as yarn-cluster then run it as a standalone
> program?
>
> Thanks
> Best Regards
>
> On Tue, Mar 17, 2015 at 1:27 AM, rrussell25 <rr...@gmail.com> wrote:
>
>> Hi, were you ever able to determine a satisfactory approach for this
>> problem?
>> I have a similar situation and would prefer to execute the job directly
>> from
>> java code within my jms listener and/or servlet container.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/What-is-best-way-to-run-spark-job-in-yarn-cluster-mode-from-java-program-servlet-container-and-NOT-u-tp21817p22086.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: What is best way to run spark job in "yarn-cluster" mode from
java program(servlet container) and NOT using spark-submit command.
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Create SparkContext set master as yarn-cluster then run it as a standalone
program?
Thanks
Best Regards
On Tue, Mar 17, 2015 at 1:27 AM, rrussell25 <rr...@gmail.com> wrote:
> Hi, were you ever able to determine a satisfactory approach for this
> problem?
> I have a similar situation and would prefer to execute the job directly
> from
> java code within my jms listener and/or servlet container.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/What-is-best-way-to-run-spark-job-in-yarn-cluster-mode-from-java-program-servlet-container-and-NOT-u-tp21817p22086.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>