You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Lin Zhao <li...@exabeam.com> on 2016/01/13 18:31:40 UTC
yarn-client: SparkSubmitDriverBootstrapper not found in yarn client
mode (1.6.0)
My job runs fine in yarn cluster mode but I have reason to use client mode instead. But I'm hitting this error when submitting:
> spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest --master yarn --deploy-mode client --executor-memory 90G --num-executors 3 --executor-cores 14 Martini-assembly-0.1.jar yarn-client
Error: Could not find or load main class org.apache.spark.deploy.SparkSubmitDriverBootstrapper
If I replace deploy-mode to cluster the job is submitted successfully. Is there a dependency missing from my project? Right now only one I included is spark-streaming 1.6.0.
Re: yarn-client: SparkSubmitDriverBootstrapper not found in yarn
client mode (1.6.0)
Posted by Lin Zhao <li...@exabeam.com>.
This is likely the cause as I replaced cdh-manager deployment (1.3.0
spark) with a compiled 1.6.0.
On 1/13/16, 10:06 AM, "Marcelo Vanzin" <va...@cloudera.com> wrote:
>SparkSubmitDriverBootstrapper was removed back in Spark 1.4, so it
>seems you have a mixbag of 1.3 / 1.6 in your path / classpath and
>things are failing because of that.
>
>On Wed, Jan 13, 2016 at 9:31 AM, Lin Zhao <li...@exabeam.com> wrote:
>> My job runs fine in yarn cluster mode but I have reason to use client
>>mode
>> instead. But I'm hitting this error when submitting:
>>> spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest
>>> --master yarn --deploy-mode client --executor-memory 90G
>>>--num-executors 3
>>> --executor-cores 14 Martini-assembly-0.1.jar yarn-client
>>
>> Error: Could not find or load main class
>> org.apache.spark.deploy.SparkSubmitDriverBootstrapper
>>
>>
>> If I replace deploy-mode to cluster the job is submitted successfully.
>>Is
>> there a dependency missing from my project? Right now only one I
>>included is
>> spark-streaming 1.6.0.
>
>
>
>--
>Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: yarn-client: SparkSubmitDriverBootstrapper not found in yarn
client mode (1.6.0)
Posted by Kevin Mellott <ke...@gmail.com>.
Lin - if you add "--verbose" to your original *spark-submit* command, it
will let you know the location in which Spark is running. As Marcelo
pointed out, this will likely indicate version 1.3, which may help you
confirm if this is your problem.
On Wed, Jan 13, 2016 at 12:06 PM, Marcelo Vanzin <va...@cloudera.com>
wrote:
> SparkSubmitDriverBootstrapper was removed back in Spark 1.4, so it
> seems you have a mixbag of 1.3 / 1.6 in your path / classpath and
> things are failing because of that.
>
> On Wed, Jan 13, 2016 at 9:31 AM, Lin Zhao <li...@exabeam.com> wrote:
> > My job runs fine in yarn cluster mode but I have reason to use client
> mode
> > instead. But I'm hitting this error when submitting:
> >> spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest
> >> --master yarn --deploy-mode client --executor-memory 90G
> --num-executors 3
> >> --executor-cores 14 Martini-assembly-0.1.jar yarn-client
> >
> > Error: Could not find or load main class
> > org.apache.spark.deploy.SparkSubmitDriverBootstrapper
> >
> >
> > If I replace deploy-mode to cluster the job is submitted successfully.
> Is
> > there a dependency missing from my project? Right now only one I
> included is
> > spark-streaming 1.6.0.
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: yarn-client: SparkSubmitDriverBootstrapper not found in yarn
client mode (1.6.0)
Posted by Marcelo Vanzin <va...@cloudera.com>.
SparkSubmitDriverBootstrapper was removed back in Spark 1.4, so it
seems you have a mixbag of 1.3 / 1.6 in your path / classpath and
things are failing because of that.
On Wed, Jan 13, 2016 at 9:31 AM, Lin Zhao <li...@exabeam.com> wrote:
> My job runs fine in yarn cluster mode but I have reason to use client mode
> instead. But I'm hitting this error when submitting:
>> spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest
>> --master yarn --deploy-mode client --executor-memory 90G --num-executors 3
>> --executor-cores 14 Martini-assembly-0.1.jar yarn-client
>
> Error: Could not find or load main class
> org.apache.spark.deploy.SparkSubmitDriverBootstrapper
>
>
> If I replace deploy-mode to cluster the job is submitted successfully. Is
> there a dependency missing from my project? Right now only one I included is
> spark-streaming 1.6.0.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: yarn-client: SparkSubmitDriverBootstrapper not found in yarn
client mode (1.6.0)
Posted by Jeff Zhang <zj...@gmail.com>.
I also didn't find SparkSubmitDriverBootstrapper, which version of spark
are you using ?
On Wed, Jan 13, 2016 at 9:36 AM, Ted Yu <yu...@gmail.com> wrote:
> Can you show the complete stack trace for the error ?
>
> I searched 1.6.0 code base but didn't find the
> class SparkSubmitDriverBootstrapper
>
> Thanks
>
> On Wed, Jan 13, 2016 at 9:31 AM, Lin Zhao <li...@exabeam.com> wrote:
>
>> My job runs fine in yarn cluster mode but I have reason to use client
>> mode instead. But I'm hitting this error when submitting:
>> > spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest
>> --master yarn --deploy-mode client --executor-memory 90G --num-executors 3
>> --executor-cores 14 Martini-assembly-0.1.jar yarn-client
>>
>> Error: Could not find or load main class
>> org.apache.spark.deploy.SparkSubmitDriverBootstrapper
>>
>> If I replace deploy-mode to cluster the job is submitted successfully.
>> Is there a dependency missing from my project? Right now only one I
>> included is spark-streaming 1.6.0.
>>
>
>
--
Best Regards
Jeff Zhang
Re: yarn-client: SparkSubmitDriverBootstrapper not found in yarn
client mode (1.6.0)
Posted by Ted Yu <yu...@gmail.com>.
Can you show the complete stack trace for the error ?
I searched 1.6.0 code base but didn't find the
class SparkSubmitDriverBootstrapper
Thanks
On Wed, Jan 13, 2016 at 9:31 AM, Lin Zhao <li...@exabeam.com> wrote:
> My job runs fine in yarn cluster mode but I have reason to use client mode
> instead. But I'm hitting this error when submitting:
> > spark-submit --class com.exabeam.martini.scripts.SparkStreamingTest
> --master yarn --deploy-mode client --executor-memory 90G --num-executors 3
> --executor-cores 14 Martini-assembly-0.1.jar yarn-client
>
> Error: Could not find or load main class
> org.apache.spark.deploy.SparkSubmitDriverBootstrapper
>
> If I replace deploy-mode to cluster the job is submitted successfully. Is
> there a dependency missing from my project? Right now only one I included
> is spark-streaming 1.6.0.
>