You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Saurav Sinha <sa...@gmail.com> on 2016/10/06 09:51:03 UTC
Detected yarn-cluster mode, but isn't running on a cluster.
Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.
Hi User,
I am trying to run spark streaming job in yarn-cluster mode.
It is failing with code
val conf = new SparkConf().setAppName("XXX"). conf.setMaster("yarn-cluster")
val ssc = new StreamingContext(conf,
Seconds(properties.getProperty("batchDurationInSeconds").toInt));
org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't
running on a cluster. Deployment to YARN is not supported directly by
SparkContext. Please use spark-submit.
at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
at
org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
at
org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" org.apache.spark.SparkException: Detected
yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is
not supported directly by SparkContext. Please use spark-submit.
at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
at
org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
at
org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Please help.
--
Thanks and Regards,
Saurav Sinha
Contact: 9742879062
Re: Detected yarn-cluster mode, but isn't running on a cluster.
Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.
Posted by Saisai Shao <sa...@gmail.com>.
You should specify cluster manager (--master) and deploy mode
(--deploy-mode) in the spark-submit arguments, specifying this through
SparkConf is too late to switch to yarn cluster mode.
On Fri, Oct 7, 2016 at 5:20 PM, Aditya <ad...@augmentiq.co.in>
wrote:
> Hi Saurav,
>
> Please share spark-submit command which you used.
>
>
> On Friday 07 October 2016 02:41 PM, Saurav Sinha wrote:
>
> I am submitting job by spark-submit but still it is giving message.
>
> Please use spark-submit.
>
> Can any one give me resone for this error.
>
> Thanks,
> Saurav Sinha
>
> On Thu, Oct 6, 2016 at 3:38 PM, Saurav Sinha <sa...@gmail.com>
> wrote:
>
>> I did not get you I am submitting job by spark-submit but still it is
>> giving message.
>>
>> Please use spark-submit.
>>
>> On Thu, Oct 6, 2016 at 3:35 PM, Sean Owen < <so...@cloudera.com>
>> sowen@cloudera.com> wrote:
>>
>>> Did you search on the internet / read the docs? the error here tells you
>>> the problem exactly. Your other question is easily answerable by looking up
>>> PermGen.
>>>
>>> On Thu, Oct 6, 2016 at 10:51 AM Saurav Sinha < <sa...@gmail.com>
>>> sauravsinha76@gmail.com> wrote:
>>>
>>>> Hi User,
>>>>
>>>> I am trying to run spark streaming job in yarn-cluster mode.
>>>>
>>>> It is failing with code
>>>>
>>>> val conf = new SparkConf().setAppName("XXX").
>>>> conf.setMaster("yarn-cluster")
>>>>
>>>> val ssc = new StreamingContext(conf, Seconds(properties.getProperty
>>>> ("batchDurationInSeconds").toInt));
>>>>
>>>>
>>>> org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't
>>>> running on a cluster. Deployment to YARN is not supported directly by
>>>> SparkContext. Please use spark-submit.
>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
>>>> at org.apache.spark.streaming.StreamingContext$.createNewSparkC
>>>> ontext(StreamingContext.scala:847)
>>>> at org.apache.spark.streaming.StreamingContext.<init>(Streaming
>>>> Context.scala:81)
>>>> at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
>>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.s
>>>> cala:215)
>>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.s
>>>> cala:215)
>>>> at scala.Option.getOrElse(Option.scala:120)
>>>> at org.apache.spark.streaming.StreamingContext$.getOrCreate(Str
>>>> eamingContext.scala:837)
>>>> at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
>>>> at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>> ssorImpl.java:62)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
>>>> $SparkSubmit$$runMain(SparkSubmit.scala:672)
>>>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit
>>>> .scala:180)
>>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>> Exception in thread "main" org.apache.spark.SparkException: Detected
>>>> yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is
>>>> not supported directly by SparkContext. Please use spark-submit.
>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
>>>> at org.apache.spark.streaming.StreamingContext$.createNewSparkC
>>>> ontext(StreamingContext.scala:847)
>>>> at org.apache.spark.streaming.StreamingContext.<init>(Streaming
>>>> Context.scala:81)
>>>> at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
>>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.s
>>>> cala:215)
>>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.s
>>>> cala:215)
>>>> at scala.Option.getOrElse(Option.scala:120)
>>>> at org.apache.spark.streaming.StreamingContext$.getOrCreate(Str
>>>> eamingContext.scala:837)
>>>> at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
>>>> at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>> ssorImpl.java:62)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
>>>> $SparkSubmit$$runMain(SparkSubmit.scala:672)
>>>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit
>>>> .scala:180)
>>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>
>>>>
>>>> Please help.
>>>> --
>>>> Thanks and Regards,
>>>>
>>>> Saurav Sinha
>>>>
>>>> Contact: 9742879062
>>>>
>>>
>>
>>
>> --
>> Thanks and Regards,
>>
>> Saurav Sinha
>>
>> Contact: 9742879062
>>
>
>
>
> --
> Thanks and Regards,
>
> Saurav Sinha
>
> Contact: 9742879062
>
>
>
>
Re: Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.
Posted by Aditya <ad...@augmentiq.co.in>.
Hi Saurav,
Please share spark-submit command which you used.
On Friday 07 October 2016 02:41 PM, Saurav Sinha wrote:
> I am submitting job by spark-submit but still it is giving message.
>
> Please use spark-submit.
>
> Can any one give me resone for this error.
>
> Thanks,
> Saurav Sinha
>
> On Thu, Oct 6, 2016 at 3:38 PM, Saurav Sinha <sauravsinha76@gmail.com
> <ma...@gmail.com>> wrote:
>
> I did not get you I am submitting job by spark-submit but still it
> is giving message.
>
> Please use spark-submit.
>
> On Thu, Oct 6, 2016 at 3:35 PM, Sean Owen <sowen@cloudera.com
> <ma...@cloudera.com>> wrote:
>
> Did you search on the internet / read the docs? the error here
> tells you the problem exactly. Your other question is easily
> answerable by looking up PermGen.
>
> On Thu, Oct 6, 2016 at 10:51 AM Saurav Sinha
> <sauravsinha76@gmail.com <ma...@gmail.com>> wrote:
>
> Hi User,
>
> I am trying to run spark streaming job in yarn-cluster mode.
>
> It is failing with code
>
> val conf = new
> SparkConf().setAppName("XXX"). conf.setMaster("yarn-cluster")
>
> val ssc = new StreamingContext(conf,
> Seconds(properties.getProperty("batchDurationInSeconds").toInt));
>
>
> org.apache.spark.SparkException: Detected yarn-cluster
> mode, but isn't running on a cluster. Deployment to YARN
> is not supported directly by SparkContext. Please use
> spark-submit.
> at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
> at
> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
> at
> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
> at
> com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
> at
> com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
> at
> com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
> at scala.Option.getOrElse(Option.scala:120)
> at
> org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
> at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
> at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
> at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
> at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Exception in thread "main"
> org.apache.spark.SparkException: Detected yarn-cluster
> mode, but isn't running on a cluster. Deployment to YARN
> is not supported directly by SparkContext. Please use
> spark-submit.
> at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
> at
> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
> at
> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
> at
> com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
> at
> com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
> at
> com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
> at scala.Option.getOrElse(Option.scala:120)
> at
> org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
> at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
> at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
> at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
> at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
> Please help.
> --
> Thanks and Regards,
>
> Saurav Sinha
>
> Contact: 9742879062
>
>
>
>
> --
> Thanks and Regards,
>
> Saurav Sinha
>
> Contact: 9742879062
>
>
>
>
> --
> Thanks and Regards,
>
> Saurav Sinha
>
> Contact: 9742879062
Re: Detected yarn-cluster mode, but isn't running on a cluster.
Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.
Posted by Saurav Sinha <sa...@gmail.com>.
I am submitting job by spark-submit but still it is giving message.
Please use spark-submit.
Can any one give me resone for this error.
Thanks,
Saurav Sinha
On Thu, Oct 6, 2016 at 3:38 PM, Saurav Sinha <sa...@gmail.com>
wrote:
> I did not get you I am submitting job by spark-submit but still it is
> giving message.
>
> Please use spark-submit.
>
> On Thu, Oct 6, 2016 at 3:35 PM, Sean Owen <so...@cloudera.com> wrote:
>
>> Did you search on the internet / read the docs? the error here tells you
>> the problem exactly. Your other question is easily answerable by looking up
>> PermGen.
>>
>> On Thu, Oct 6, 2016 at 10:51 AM Saurav Sinha <sa...@gmail.com>
>> wrote:
>>
>>> Hi User,
>>>
>>> I am trying to run spark streaming job in yarn-cluster mode.
>>>
>>> It is failing with code
>>>
>>> val conf = new SparkConf().setAppName("XXX").
>>> conf.setMaster("yarn-cluster")
>>>
>>> val ssc = new StreamingContext(conf, Seconds(properties.getProperty
>>> ("batchDurationInSeconds").toInt));
>>>
>>>
>>> org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't
>>> running on a cluster. Deployment to YARN is not supported directly by
>>> SparkContext. Please use spark-submit.
>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
>>> at org.apache.spark.streaming.StreamingContext$.createNewSparkC
>>> ontext(StreamingContext.scala:847)
>>> at org.apache.spark.streaming.StreamingContext.<init>(Streaming
>>> Context.scala:81)
>>> at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at org.apache.spark.streaming.StreamingContext$.getOrCreate(Str
>>> eamingContext.scala:837)
>>> at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
>>> at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
>>> $SparkSubmit$$runMain(SparkSubmit.scala:672)
>>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit
>>> .scala:180)
>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>> Exception in thread "main" org.apache.spark.SparkException: Detected
>>> yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is
>>> not supported directly by SparkContext. Please use spark-submit.
>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
>>> at org.apache.spark.streaming.StreamingContext$.createNewSparkC
>>> ontext(StreamingContext.scala:847)
>>> at org.apache.spark.streaming.StreamingContext.<init>(Streaming
>>> Context.scala:81)
>>> at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>>> at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at org.apache.spark.streaming.StreamingContext$.getOrCreate(Str
>>> eamingContext.scala:837)
>>> at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
>>> at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
>>> $SparkSubmit$$runMain(SparkSubmit.scala:672)
>>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit
>>> .scala:180)
>>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>>
>>> Please help.
>>> --
>>> Thanks and Regards,
>>>
>>> Saurav Sinha
>>>
>>> Contact: 9742879062
>>>
>>
>
>
> --
> Thanks and Regards,
>
> Saurav Sinha
>
> Contact: 9742879062
>
--
Thanks and Regards,
Saurav Sinha
Contact: 9742879062