You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Philip Ogren <ph...@oracle.com> on 2013/11/23 03:39:36 UTC

akka config settings

[Note: I answered my own question while writing this up and thought I 
would share my findings anyways since it was a rather painful discovery 
process.]

I get the following error when trying to run Spark in standalone mode 
when I package up my code in a jar file:

No configuration setting found for key 'akka.remote.log-received-messages'

I believe this is exactly the same issue that is discussed here:

https://groups.google.com/forum/#!topic/spark-users/FmYGAP57MMk 
<https://groups.google.com/forum/#%21topic/spark-users/FmYGAP57MMk>

I followed this thread closely, updated my pom.xml accordingly, and 
filled in a couple of details that are not explicit from the thread.  In 
particular, I added the filter tag as described in here 
<https://www.assembla.com/spaces/akka/tickets/2634-akka-remote-prevents-application-from--shading--%28standalone-jar-creation%29-due-to-config-issues-in---?comment=183467573#comment:183467573> 
(as linked from the thread.)  Also, (now obvious) step of adding a 
reference.conf file to my project's src/main/resources directory so that 
it would get picked up.  My problem was that I couldn't figure out where 
to find the reference.conf file!  I tried a number of things including 
extracting example reference.conf files out of various akka jar files in 
my local maven repo.  Short story long, I did a google search on "spark 
reference.conf" and came across this:

https://spark-project.atlassian.net/browse/SPARK-395

which led me to this:

http://letitcrash.com/post/21025950392/howto-sbt-assembly-vs-reference-conf

which led me to realize that I needed to pull the reference.conf out of 
spark-assembly-0.8.0-incubating-hadoop2.0.0-cdh4.3.0.jar.  It's right 
there!

Summary - a good akka reference.conf file can be found in the top level 
of spark-assembly-0.8.0-incubating-hadoop2.0.0-cdh4.3.0.jar and can be 
placed in src/main/resources before packaging up your application.  My 
problems went away....

Any thoughts about how this could be simplified?

HTH!
Philip