You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Michael Lewis <le...@icloud.com> on 2015/10/23 01:39:41 UTC

Spark issue running jar on Linux vs Windows

Hi,

I have a Spark driver process that I have built into a single ‘fat jar’ this runs fine,  in Cygwin, on my development machine,
I can run:
 
scala -cp my-fat-jar-1.0.0.jar com.foo.MyMainClass

this works fine, it will submit Spark job, they process, all good.


However, on Linux (all Jars Spark(1.4) and Scala version (2.10.5) being the same), I get this error:

18:59:14.358 [Curator-QueueBuilder-2] ERROR o.a.c.f.r.queue.DistributedQueue - Exception processing queue item: queue-0000000000
java.lang.NoSuchMethodException: akka.remote.RemoteActorRefProvider.<init>(java.lang.String, akka.actor.ActorSystem$Settings, akka.event.EventStream, akka.actor.Scheduler, akka.act
        at java.lang.Class.getConstructor0(Class.java:3082) ~[na:1.8.0_60]
        at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[na:1.8.0_60]
 
i.e. No such method exception.  Can anyone suggest how I can fix this?

I’ve tried changing the scala to java and putting scale-lang on the class path, but this just generates new errors about missing akka configuration.

Given the identical jars and scala version - I’m not sure why I’m getting this error running driver on Linux.

Appreciate any help/pointers.


Thanks,
Mike Lewis

Re: Spark issue running jar on Linux vs Windows

Posted by Ted Yu <yu...@gmail.com>.
RemoteActorRefProvider is in akka-remote_2.10-2.3.11.jar

jar tvf
~/.m2/repository/com/typesafe/akka/akka-remote_2.10/2.3.11/akka-remote_2.10-2.3.11.jar
| grep RemoteActorRefProvi
  1761 Fri May 08 16:13:02 PDT 2015
akka/remote/RemoteActorRefProvider$$anonfun$5.class
  1416 Fri May 08 16:13:02 PDT 2015
akka/remote/RemoteActorRefProvider$$anonfun$6.class

Is the above jar on your classpath ?

Cheers

On Thu, Oct 22, 2015 at 4:39 PM, Michael Lewis <le...@icloud.com> wrote:

> Hi,
>
> I have a Spark driver process that I have built into a single ‘fat jar’
> this runs fine,  in Cygwin, on my development machine,
> I can run:
>
> scala -cp my-fat-jar-1.0.0.jar com.foo.MyMainClass
>
> this works fine, it will submit Spark job, they process, all good.
>
>
> However, on Linux (all Jars Spark(1.4) and Scala version (2.10.5) being
> the same), I get this error:
>
> 18:59:14.358 [Curator-QueueBuilder-2] ERROR
> o.a.c.f.r.queue.DistributedQueue - Exception processing queue item:
> queue-0000000000
> java.lang.NoSuchMethodException:
> akka.remote.RemoteActorRefProvider.<init>(java.lang.String,
> akka.actor.ActorSystem$Settings, akka.event.EventStream,
> akka.actor.Scheduler, akka.act
>         at java.lang.Class.getConstructor0(Class.java:3082) ~[na:1.8.0_60]
>         at java.lang.Class.getDeclaredConstructor(Class.java:2178)
> ~[na:1.8.0_60]
>
>
> i.e. No such method exception.  Can anyone suggest how I can fix this?
>
> I’ve tried changing the scala to java and putting scale-lang on the class
> path, but this just generates new errors about missing akka configuration.
>
> Given the identical jars and scala version - I’m not sure why I’m getting
> this error running driver on Linux.
>
> Appreciate any help/pointers.
>
>
> Thanks,
> Mike Lewis
>

Re: Spark issue running jar on Linux vs Windows

Posted by Michael Lewis <le...@icloud.com>.
Thanks for the advice. In my case it turned out to be two issues.

- use Java rather than Scala to launch the process, putting the core Scala libs on the class path.

- I needed a merge strategy of Concat for reference.conf files in my build.sbt

Regards,
Mike


> On 23 Oct 2015, at 01:00, Ted Yu <yu...@gmail.com> wrote:
> 
> RemoteActorRefProvider is in akka-remote_2.10-2.3.11.jar
> 
> jar tvf ~/.m2/repository/com/typesafe/akka/akka-remote_2.10/2.3.11/akka-remote_2.10-2.3.11.jar | grep RemoteActorRefProvi
>   1761 Fri May 08 16:13:02 PDT 2015 akka/remote/RemoteActorRefProvider$$anonfun$5.class
>   1416 Fri May 08 16:13:02 PDT 2015 akka/remote/RemoteActorRefProvider$$anonfun$6.class
> 
> Is the above jar on your classpath ?
> 
> Cheers
> 
>> On Thu, Oct 22, 2015 at 4:39 PM, Michael Lewis <le...@icloud.com> wrote:
>> Hi,
>> 
>> I have a Spark driver process that I have built into a single ‘fat jar’ this runs fine,  in Cygwin, on my development machine,
>> I can run:
>>  
>> scala -cp my-fat-jar-1.0.0.jar com.foo.MyMainClass
>> 
>> this works fine, it will submit Spark job, they process, all good.
>> 
>> 
>> However, on Linux (all Jars Spark(1.4) and Scala version (2.10.5) being the same), I get this error:
>> 
>> 18:59:14.358 [Curator-QueueBuilder-2] ERROR o.a.c.f.r.queue.DistributedQueue - Exception processing queue item: queue-0000000000
>> java.lang.NoSuchMethodException: akka.remote.RemoteActorRefProvider.<init>(java.lang.String, akka.actor.ActorSystem$Settings, akka.event.EventStream, akka.actor.Scheduler, akka.act
>>         at java.lang.Class.getConstructor0(Class.java:3082) ~[na:1.8.0_60]
>>         at java.lang.Class.getDeclaredConstructor(Class.java:2178) ~[na:1.8.0_60]
>>  
>> i.e. No such method exception.  Can anyone suggest how I can fix this?
>> 
>> I’ve tried changing the scala to java and putting scale-lang on the class path, but this just generates new errors about missing akka configuration.
>> 
>> Given the identical jars and scala version - I’m not sure why I’m getting this error running driver on Linux.
>> 
>> Appreciate any help/pointers.
>> 
>> 
>> Thanks,
>> Mike Lewis
> 

Re: Spark issue running jar on Linux vs Windows

Posted by Ted Yu <yu...@gmail.com>.
RemoteActorRefProvider is in akka-remote_2.10-2.3.11.jar

jar tvf
~/.m2/repository/com/typesafe/akka/akka-remote_2.10/2.3.11/akka-remote_2.10-2.3.11.jar
| grep RemoteActorRefProvi
  1761 Fri May 08 16:13:02 PDT 2015
akka/remote/RemoteActorRefProvider$$anonfun$5.class
  1416 Fri May 08 16:13:02 PDT 2015
akka/remote/RemoteActorRefProvider$$anonfun$6.class

Is the above jar on your classpath ?

Cheers

On Thu, Oct 22, 2015 at 4:39 PM, Michael Lewis <le...@icloud.com> wrote:

> Hi,
>
> I have a Spark driver process that I have built into a single ‘fat jar’
> this runs fine,  in Cygwin, on my development machine,
> I can run:
>
> scala -cp my-fat-jar-1.0.0.jar com.foo.MyMainClass
>
> this works fine, it will submit Spark job, they process, all good.
>
>
> However, on Linux (all Jars Spark(1.4) and Scala version (2.10.5) being
> the same), I get this error:
>
> 18:59:14.358 [Curator-QueueBuilder-2] ERROR
> o.a.c.f.r.queue.DistributedQueue - Exception processing queue item:
> queue-0000000000
> java.lang.NoSuchMethodException:
> akka.remote.RemoteActorRefProvider.<init>(java.lang.String,
> akka.actor.ActorSystem$Settings, akka.event.EventStream,
> akka.actor.Scheduler, akka.act
>         at java.lang.Class.getConstructor0(Class.java:3082) ~[na:1.8.0_60]
>         at java.lang.Class.getDeclaredConstructor(Class.java:2178)
> ~[na:1.8.0_60]
>
>
> i.e. No such method exception.  Can anyone suggest how I can fix this?
>
> I’ve tried changing the scala to java and putting scale-lang on the class
> path, but this just generates new errors about missing akka configuration.
>
> Given the identical jars and scala version - I’m not sure why I’m getting
> this error running driver on Linux.
>
> Appreciate any help/pointers.
>
>
> Thanks,
> Mike Lewis
>