You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Dwane Hall <dw...@hotmail.com> on 2019/07/12 02:50:59 UTC

Spark-Solr connector

Hey guys,



I’ve just started looking at the excellent spark-solr project (thanks Tim Potter, Kiran Chitturi, Kevin Risden and Jason Gerlowski for their efforts with this project it looks really neat!!).



I’m only at the initial stages of my exploration but I’m running into a class not found exception when connecting to a secure solr cloud instance (basic auth, ssl).  Everything is working as expected on a non-secure solr cloud instance.



The process looks pretty straightforward according to the doco so I’m wondering if I’m missing anything obvious or if I need to bring any extra classes to the classpath when using this project?



Any advice would be greatly appreciated.



Thanks,



Dwane



Environments tried

7.6 and 8.1.1 solr cloud

SSL, Basic Auth Plugin, Rules Based Authorisation Plugin enabled

Spark v 2.4.3

Spark-Solr build spark-solr-3.7.0-20190619.153847-16-shaded.jar





./spark-2.4.3-bin-hado./spark-2.4.3-bin-hadoop2.7/bin/spark-shell --master local[*] --jars spark-solr-3.7.0-20190619.153847-16-shaded.jar --conf 'spark.driver.extraJavaOptions=-Dbasicauth=solr:SolrRocks'





val options = Map(

        "collection" -> "My_Collection",

        "zkhost" -> "zkn1:2181,zkn2:2181,zkn3:2181/solr/SPARKTEST"

      )



val df = spark.read.format("solr").options(options).load



com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: org/eclipse/jetty/client/api/Authentication

  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2261)

  at com.google.common.cache.LocalCache.get(LocalCache.java:4000)

  at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)

  at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)

  at com.lucidworks.spark.util.SolrSupport$.getCachedCloudClient(SolrSupport.scala:244)

  at com.lucidworks.spark.util.SolrSupport$.getSolrBaseUrl(SolrSupport.scala:248)

  at com.lucidworks.spark.SolrRelation.dynamicSuffixes$lzycompute(SolrRelation.scala:100)

  at com.lucidworks.spark.SolrRelation.dynamicSuffixes(SolrRelation.scala:98)

  at com.lucidworks.spark.SolrRelation.getBaseSchemaFromConfig(SolrRelation.scala:299)

  at com.lucidworks.spark.SolrRelation.querySchema$lzycompute(SolrRelation.scala:239)

  at com.lucidworks.spark.SolrRelation.querySchema(SolrRelation.scala:108)

  at com.lucidworks.spark.SolrRelation.schema(SolrRelation.scala:428)

  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:403)

  at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)

  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)

  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)

  ... 49 elided

Caused by: java.lang.NoClassDefFoundError: org/eclipse/jetty/client/api/Authentication

  at com.lucidworks.spark.util.SolrSupport$.getSolrCloudClient(SolrSupport.scala:214)

  at com.lucidworks.spark.util.SolrSupport$.getNewSolrCloudClient(SolrSupport.scala:240)

  at com.lucidworks.spark.util.CacheCloudSolrClient$$anon$1.load(SolrSupport.scala:38)

  at com.lucidworks.spark.util.CacheCloudSolrClient$$anon$1.load(SolrSupport.scala:36)

  at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)

  at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)

  at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)

  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)

  ... 64 more

Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.client.api.Authentication

  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

  ... 72 more

Re: Spark-Solr connector

Posted by Dwane Hall <dw...@hotmail.com>.

Thanks Shawn I'll raise a question on the GitHub page. Cheers,
Dwane
________________________________
From: Shawn Heisey <ap...@elyograg.org>
Sent: Friday, 12 July 2019 10:05 PM
To: solr-user@lucene.apache.org
Subject: Re: Spark-Solr connector

On 7/11/2019 8:50 PM, Dwane Hall wrote:
> I’ve just started looking at the excellent spark-solr project (thanks Tim Potter, Kiran Chitturi, Kevin Risden and Jason Gerlowski for their efforts with this project it looks really neat!!).
>
> I’m only at the initial stages of my exploration but I’m running into a class not found exception when connecting to a secure solr cloud instance (basic auth, ssl).  Everything is working as expected on a non-secure solr cloud instance.
>
> The process looks pretty straightforward according to the doco so I’m wondering if I’m missing anything obvious or if I need to bring any extra classes to the classpath when using this project?
>
> Any advice would be greatly appreciated.

The exception here (which I did not quote) is in code from Google,
Spark, and Lucidworks.  There are no Solr classes mentioned at all in
the stacktrace.

Which means that we won't be able to help you on this list.  Looking
closer at the stacktrace, it looks to me like you're going to need to
talk to Lucidworks about this problem.

Thanks,
Shawn

Re: Spark-Solr connector

Posted by Shawn Heisey <ap...@elyograg.org>.

On 7/11/2019 8:50 PM, Dwane Hall wrote:
> I’ve just started looking at the excellent spark-solr project (thanks Tim Potter, Kiran Chitturi, Kevin Risden and Jason Gerlowski for their efforts with this project it looks really neat!!).
> 
> I’m only at the initial stages of my exploration but I’m running into a class not found exception when connecting to a secure solr cloud instance (basic auth, ssl).  Everything is working as expected on a non-secure solr cloud instance.
> 
> The process looks pretty straightforward according to the doco so I’m wondering if I’m missing anything obvious or if I need to bring any extra classes to the classpath when using this project?
> 
> Any advice would be greatly appreciated.

The exception here (which I did not quote) is in code from Google, 
Spark, and Lucidworks.  There are no Solr classes mentioned at all in 
the stacktrace.

Which means that we won't be able to help you on this list.  Looking 
closer at the stacktrace, it looks to me like you're going to need to 
talk to Lucidworks about this problem.

Thanks,
Shawn