Error starting solr with plugin which uses spark libraries

[Rahul Kumar <ra...@snapdeal.com>]

Hi all,
I am trying to start solr with a custom plugin which uses spark library. I
am trying to initialize sparkcontext in local mode. I have made a fat jar
for this plugin using maven shade and put it in the lib directory. *While
starting solr it is not able to initialize sparkcontext.* It says class not
found exception for AkkaRpcEnvFactory. Can anyone please help.

*It gives the following error:*

3870 [coreLoadExecutor-4-thread-1] ERROR org.apache.spark.SparkContext
 – Error initializing SparkContext.
java.lang.ClassNotFoundException:org.apache.spark.rpc.akka.AkkaRpcEnvFactory

*Here is the detailed error*

java -jar start.jar0    [main] INFO  org.eclipse.jetty.server.Server
– jetty-8.1.10.v2013031227   [main] INFO
org.eclipse.jetty.deploy.providers.ScanningAppProvider  – Deployment
monitor /home/rahul/solr-4.7.2/example/contexts at interval 040
[main] INFO  org.eclipse.jetty.deploy.DeploymentManager  – Deployable
added: /home/rahul/solr-4.7.2/example/contexts/solr-jetty-context.xml1095
[main] INFO  org.eclipse.jetty.webapp.StandardDescriptorProcessor  –
NO JSP Support for /solr, did not find
org.apache.jasper.servlet.JspServlet1155 [main] INFO
org.apache.solr.servlet.SolrDispatchFilter  –
SolrDispatchFilter.init()1189 [main] INFO
org.apache.solr.core.SolrResourceLoader  – JNDI not configured for
solr (NoInitialContextEx)1190 [main] INFO
org.apache.solr.core.SolrResourceLoader  – solr home defaulted to
'solr/' (could not find system property or JNDI)1190 [main] INFO
org.apache.solr.core.SolrResourceLoader  – new SolrResourceLoader for
directory: 'solr/'1280 [main] INFO  org.apache.solr.core.ConfigSolr  –
Loading container configuration from
/home/rahul/solr-4.7.2/example/solr/solr.xml1458 [main] INFO
org.apache.solr.core.CoresLocator  – Config-defined core root
directory: /home/rahul/solr-4.7.2/example/solr1465 [main] INFO
org.apache.solr.core.CoreContainer  – New CoreContainer
602710225...............
3870 [coreLoadExecutor-4-thread-1] ERROR org.apache.spark.SparkContext
 – Error initializing SparkContext.
java.lang.ClassNotFoundException: org.apache.spark.rpc.akka.AkkaRpcEnvFactory
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:430)
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:383)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:274)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
    at org.apache.spark.rpc.RpcEnv$.getRpcEnvFactory(RpcEnv.scala:42)
    at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:53)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:252)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:193)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:276)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:441)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
    at com.snapdeal.search.spark.SparkLoadModel.loadModel(SparkLoadModel.java:11)
    at com.snapdeal.search.valuesource.parser.RankingModelValueSourceParser.init(RankingModelValueSourceParser.java:29)
    at org.apache.solr.core.SolrCore.createInitInstance(SolrCore.java:591)
    at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2191)
    at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2185)
    at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:2218)
    at org.apache.solr.core.SolrCore.initValueSourceParsers(SolrCore.java:2130)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:765)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:630)
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:562)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:597)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:258)
    at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:250)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)3880
[coreLoadExecutor-4-thread-1] INFO  org.apache.spark.SparkContext  –
Successfully stopped SparkContext





Rahul Kumar
*Software Engineer- I (Search Snapdeal)*

*M*: +91 9023542950    *EXT: *14226
362-363, ASF CENTRE , UDYOG VIHAR , PHASE - IV , GURGAON 122 016 , INDIA