You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Michael Lewis <le...@icloud.com> on 2015/10/23 01:39:41 UTC
Spark issue running jar on Linux vs Windows
Hi,
I have a Spark driver process that I have built into a single ‘fat jar’ this runs fine, in Cygwin, on my development machine,
I can run:
scala -cp my-fat-jar-1.0.0.jar com.foo.MyMainClass
this works fine, it will submit Spark job, they process, all good.
However, on Linux (all Jars Spark(1.4) and Scala version (2.10.5) being the same), I get this error:
18:59:14.358 [Curator-QueueBuilder-2] ERROR o.a.c.f.r.queue.DistributedQueue - Exception processing queue item: queue-0000000000
java.lang.NoSuchMethodException: akka.remote.RemoteActorRefProvider.<init>(java.lang.String, akka.actor.ActorSystem$Settings, akka.event.EventStream, akka.actor.Scheduler, akka.act
at java.lang.Class.getConstructor0(Class.java:3082) ~[na:1.8.0_60]
at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[na:1.8.0_60]
i.e. No such method exception. Can anyone suggest how I can fix this?
I’ve tried changing the scala to java and putting scale-lang on the class path, but this just generates new errors about missing akka configuration.
Given the identical jars and scala version - I’m not sure why I’m getting this error running driver on Linux.
Appreciate any help/pointers.
Thanks,
Mike Lewis
Re: Spark issue running jar on Linux vs Windows
Posted by Ted Yu <yu...@gmail.com>.
RemoteActorRefProvider is in akka-remote_2.10-2.3.11.jar
jar tvf
~/.m2/repository/com/typesafe/akka/akka-remote_2.10/2.3.11/akka-remote_2.10-2.3.11.jar
| grep RemoteActorRefProvi
1761 Fri May 08 16:13:02 PDT 2015
akka/remote/RemoteActorRefProvider$$anonfun$5.class
1416 Fri May 08 16:13:02 PDT 2015
akka/remote/RemoteActorRefProvider$$anonfun$6.class
Is the above jar on your classpath ?
Cheers
On Thu, Oct 22, 2015 at 4:39 PM, Michael Lewis <le...@icloud.com> wrote:
> Hi,
>
> I have a Spark driver process that I have built into a single ‘fat jar’
> this runs fine, in Cygwin, on my development machine,
> I can run:
>
> scala -cp my-fat-jar-1.0.0.jar com.foo.MyMainClass
>
> this works fine, it will submit Spark job, they process, all good.
>
>
> However, on Linux (all Jars Spark(1.4) and Scala version (2.10.5) being
> the same), I get this error:
>
> 18:59:14.358 [Curator-QueueBuilder-2] ERROR
> o.a.c.f.r.queue.DistributedQueue - Exception processing queue item:
> queue-0000000000
> java.lang.NoSuchMethodException:
> akka.remote.RemoteActorRefProvider.<init>(java.lang.String,
> akka.actor.ActorSystem$Settings, akka.event.EventStream,
> akka.actor.Scheduler, akka.act
> at java.lang.Class.getConstructor0(Class.java:3082) ~[na:1.8.0_60]
> at java.lang.Class.getDeclaredConstructor(Class.java:2178)
> ~[na:1.8.0_60]
>
>
> i.e. No such method exception. Can anyone suggest how I can fix this?
>
> I’ve tried changing the scala to java and putting scale-lang on the class
> path, but this just generates new errors about missing akka configuration.
>
> Given the identical jars and scala version - I’m not sure why I’m getting
> this error running driver on Linux.
>
> Appreciate any help/pointers.
>
>
> Thanks,
> Mike Lewis
>
Re: Spark issue running jar on Linux vs Windows
Posted by Michael Lewis <le...@icloud.com>.
Thanks for the advice. In my case it turned out to be two issues.
- use Java rather than Scala to launch the process, putting the core Scala libs on the class path.
- I needed a merge strategy of Concat for reference.conf files in my build.sbt
Regards,
Mike
> On 23 Oct 2015, at 01:00, Ted Yu <yu...@gmail.com> wrote:
>
> RemoteActorRefProvider is in akka-remote_2.10-2.3.11.jar
>
> jar tvf ~/.m2/repository/com/typesafe/akka/akka-remote_2.10/2.3.11/akka-remote_2.10-2.3.11.jar | grep RemoteActorRefProvi
> 1761 Fri May 08 16:13:02 PDT 2015 akka/remote/RemoteActorRefProvider$$anonfun$5.class
> 1416 Fri May 08 16:13:02 PDT 2015 akka/remote/RemoteActorRefProvider$$anonfun$6.class
>
> Is the above jar on your classpath ?
>
> Cheers
>
>> On Thu, Oct 22, 2015 at 4:39 PM, Michael Lewis <le...@icloud.com> wrote:
>> Hi,
>>
>> I have a Spark driver process that I have built into a single ‘fat jar’ this runs fine, in Cygwin, on my development machine,
>> I can run:
>>
>> scala -cp my-fat-jar-1.0.0.jar com.foo.MyMainClass
>>
>> this works fine, it will submit Spark job, they process, all good.
>>
>>
>> However, on Linux (all Jars Spark(1.4) and Scala version (2.10.5) being the same), I get this error:
>>
>> 18:59:14.358 [Curator-QueueBuilder-2] ERROR o.a.c.f.r.queue.DistributedQueue - Exception processing queue item: queue-0000000000
>> java.lang.NoSuchMethodException: akka.remote.RemoteActorRefProvider.<init>(java.lang.String, akka.actor.ActorSystem$Settings, akka.event.EventStream, akka.actor.Scheduler, akka.act
>> at java.lang.Class.getConstructor0(Class.java:3082) ~[na:1.8.0_60]
>> at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[na:1.8.0_60]
>>
>> i.e. No such method exception. Can anyone suggest how I can fix this?
>>
>> I’ve tried changing the scala to java and putting scale-lang on the class path, but this just generates new errors about missing akka configuration.
>>
>> Given the identical jars and scala version - I’m not sure why I’m getting this error running driver on Linux.
>>
>> Appreciate any help/pointers.
>>
>>
>> Thanks,
>> Mike Lewis
>
Re: Spark issue running jar on Linux vs Windows
Posted by Ted Yu <yu...@gmail.com>.
RemoteActorRefProvider is in akka-remote_2.10-2.3.11.jar
jar tvf
~/.m2/repository/com/typesafe/akka/akka-remote_2.10/2.3.11/akka-remote_2.10-2.3.11.jar
| grep RemoteActorRefProvi
1761 Fri May 08 16:13:02 PDT 2015
akka/remote/RemoteActorRefProvider$$anonfun$5.class
1416 Fri May 08 16:13:02 PDT 2015
akka/remote/RemoteActorRefProvider$$anonfun$6.class
Is the above jar on your classpath ?
Cheers
On Thu, Oct 22, 2015 at 4:39 PM, Michael Lewis <le...@icloud.com> wrote:
> Hi,
>
> I have a Spark driver process that I have built into a single ‘fat jar’
> this runs fine, in Cygwin, on my development machine,
> I can run:
>
> scala -cp my-fat-jar-1.0.0.jar com.foo.MyMainClass
>
> this works fine, it will submit Spark job, they process, all good.
>
>
> However, on Linux (all Jars Spark(1.4) and Scala version (2.10.5) being
> the same), I get this error:
>
> 18:59:14.358 [Curator-QueueBuilder-2] ERROR
> o.a.c.f.r.queue.DistributedQueue - Exception processing queue item:
> queue-0000000000
> java.lang.NoSuchMethodException:
> akka.remote.RemoteActorRefProvider.<init>(java.lang.String,
> akka.actor.ActorSystem$Settings, akka.event.EventStream,
> akka.actor.Scheduler, akka.act
> at java.lang.Class.getConstructor0(Class.java:3082) ~[na:1.8.0_60]
> at java.lang.Class.getDeclaredConstructor(Class.java:2178)
> ~[na:1.8.0_60]
>
>
> i.e. No such method exception. Can anyone suggest how I can fix this?
>
> I’ve tried changing the scala to java and putting scale-lang on the class
> path, but this just generates new errors about missing akka configuration.
>
> Given the identical jars and scala version - I’m not sure why I’m getting
> this error running driver on Linux.
>
> Appreciate any help/pointers.
>
>
> Thanks,
> Mike Lewis
>