You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by GitBox <gi...@apache.org> on 2020/04/24 11:49:44 UTC
[GitHub] [nutch] balashashanka opened a new pull request #515: NUTCH-2780 : Upgrade index-solr to use Solr 8.5.1
balashashanka opened a new pull request #515:
URL: https://github.com/apache/nutch/pull/515
Upgraded the solr client to 8.5.1
Registered all the dependencies in plugin.xml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on pull request #515: NUTCH-2780 : Upgrade index-solr to use Solr 8.5.1
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on pull request #515:
URL: https://github.com/apache/nutch/pull/515#issuecomment-619975949
Hi @balashashanka, you'll find all fixes in c6646fd. Ok, I forgot to mention that the slf4j-api dependency needs also be removed from the plugin.xml. Don't know whether the ivy version makes a difference, but we use ivy 2.4.0 in core, so we should use it also for the plugins. For testing, I've installed Solr 8.5.1 as described in the [Nutch tutorial](https://cwiki.apache.org/confluence/display/NUTCH/NutchTutorial#NutchTutorial-SetupSolrforsearch) and used the tool indexchecker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] balashashanka commented on pull request #515: NUTCH-2780 : Upgrade index-solr to use Solr 8.5.1
Posted by GitBox <gi...@apache.org>.
balashashanka commented on pull request #515:
URL: https://github.com/apache/nutch/pull/515#issuecomment-619984565
Removing the sl4j-api fixed it for me thanks. I've added the howto_upgrade_solr.txt with the reference from the elastic [howto](https://github.com/apache/nutch/blob/master/src/plugin/indexer-elastic/howto_upgrade_es.txt). Will test the changes and commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] balashashanka commented on a change in pull request #515: NUTCH-2780 : Upgrade index-solr to use Solr 8.5.1
Posted by GitBox <gi...@apache.org>.
balashashanka commented on a change in pull request #515:
URL: https://github.com/apache/nutch/pull/515#discussion_r415770564
##########
File path: src/plugin/indexer-solr/ivy.xml
##########
@@ -36,9 +36,9 @@
</publications>
<dependencies>
- <dependency org="org.apache.solr" name="solr-solrj" rev="7.3.1"/>
- <dependency org="org.apache.httpcomponents" name="httpcore" rev="4.4.6" conf="*->default"/>
- <dependency org="org.apache.httpcomponents" name="httpmime" rev="4.5.3" conf="*->default"/>
+ <dependency org="org.apache.solr" name="solr-solrj" rev="8.5.1"/>
+ <dependency org="org.apache.httpcomponents" name="httpcore" rev="4.4.12" conf="*->default"/>
+ <dependency org="org.apache.httpcomponents" name="httpmime" rev="4.5.10" conf="*->default"/>
Review comment:
Hi @sebastian-nagel , i tried fixing this but with no positive results. Could you tell me how you fixed it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel edited a comment on pull request #515: NUTCH-2780 : Upgrade index-solr to use Solr 8.5.1
Posted by GitBox <gi...@apache.org>.
sebastian-nagel edited a comment on pull request #515:
URL: https://github.com/apache/nutch/pull/515#issuecomment-619975949
Hi @balashashanka, you'll find all fixes in c6646fd. Ok, I forgot to mention that the slf4j-api dependency needs also be removed from the plugin.xml. Don't know whether the ivy version makes a difference, but we use ivy 2.4.0 in core, so we should use it also for the plugins. For testing, I've installed Solr 8.5.1 as described in the [Nutch tutorial](https://cwiki.apache.org/confluence/display/NUTCH/NutchTutorial#NutchTutorial-SetupSolrforsearch) and used the tool indexchecker. And don't forget to clean and rebuild by `ant clean runtime` after the changes have been applied.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on a change in pull request #515: NUTCH-2780 : Upgrade index-solr to use Solr 8.5.1
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on a change in pull request #515:
URL: https://github.com/apache/nutch/pull/515#discussion_r415537853
##########
File path: src/plugin/indexer-solr/ivy.xml
##########
@@ -36,9 +36,9 @@
</publications>
<dependencies>
- <dependency org="org.apache.solr" name="solr-solrj" rev="7.3.1"/>
- <dependency org="org.apache.httpcomponents" name="httpcore" rev="4.4.6" conf="*->default"/>
- <dependency org="org.apache.httpcomponents" name="httpmime" rev="4.5.3" conf="*->default"/>
+ <dependency org="org.apache.solr" name="solr-solrj" rev="8.5.1"/>
+ <dependency org="org.apache.httpcomponents" name="httpcore" rev="4.4.12" conf="*->default"/>
+ <dependency org="org.apache.httpcomponents" name="httpmime" rev="4.5.10" conf="*->default"/>
Review comment:
Should be httpmime 4.5.10 (httpcore 4.4.12), otherwise you'll see:
```
$> bin/nutch indexchecker -Dplugin.includes='protocol-okhttp|parse-tika|index-(basic|more|anchor)|indexer-solr' \
-doIndex https://localhost/
...
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/entity/mime/content/ContentBody
at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:952)
at org.apache.nutch.indexwriter.solr.SolrUtils.getHttpSolrClient(SolrUtils.java:59)
at org.apache.nutch.indexwriter.solr.SolrIndexWriter.open(SolrIndexWriter.java:107)
at org.apache.nutch.indexer.IndexWriters.open(IndexWriters.java:216)
at org.apache.nutch.indexer.IndexingFiltersChecker.process(IndexingFiltersChecker.java:296)
at org.apache.nutch.util.AbstractChecker.processSingle(AbstractChecker.java:85)
at org.apache.nutch.indexer.IndexingFiltersChecker.run(IndexingFiltersChecker.java:132)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.nutch.indexer.IndexingFiltersChecker.main(IndexingFiltersChecker.java:305)
Caused by: java.lang.ClassNotFoundException: org.apache.http.entity.mime.content.ContentBody
```
##########
File path: src/plugin/indexer-solr/plugin.xml
##########
@@ -22,15 +22,36 @@
<library name="indexer-solr.jar">
<export name="*" />
</library>
- <library name="commons-io-2.5.jar"/>
- <library name="httpclient-4.5.3.jar"/>
+ <library name="commons-io-2.6.jar"/>
+ <library name="netty-buffer-4.1.29.Final.jar"/>
+ <library name="netty-codec-4.1.29.Final.jar"/>
+ <library name="netty-common-4.1.29.Final.jar"/>
+ <library name="netty-handler-4.1.29.Final.jar"/>
+ <library name="netty-resolver-4.1.29.Final.jar"/>
+ <library name="netty-transport-4.1.29.Final.jar"/>
+ <library name="netty-transport-native-epoll-4.1.29.Final.jar"/>
+ <library name="netty-transport-native-unix-common-4.1.29.Final.jar"/>
+ <library name="commons-math3-3.6.1.jar"/>
+ <library name="httpclient-4.5.10.jar"/>
<library name="httpcore-4.4.6.jar"/>
<library name="httpmime-4.5.3.jar"/>
- <library name="noggit-0.8.jar"/>
- <library name="solr-solrj-7.3.1.jar"/>
+ <library name="zookeeper-3.5.5.jar"/>
+ <library name="zookeeper-jute-3.5.5.jar"/>
<library name="stax2-api-3.1.4.jar"/>
<library name="woodstox-core-asl-4.4.1.jar"/>
- <library name="zookeeper-3.4.11.jar"/>
+ <library name="jetty-alpn-client-9.4.24.v20191120.jar"/>
+ <library name="jetty-alpn-java-client-9.4.24.v20191120.jar"/>
+ <library name="jetty-client-9.4.24.v20191120.jar"/>
+ <library name="jetty-http-9.4.24.v20191120.jar"/>
+ <library name="jetty-io-9.4.24.v20191120.jar"/>
+ <library name="jetty-util-9.4.24.v20191120.jar"/>
+ <library name="http2-client-9.4.24.v20191120.jar"/>
+ <library name="http2-common-9.4.24.v20191120.jar"/>
+ <library name="http2-hpack-9.4.24.v20191120.jar"/>
+ <library name="http2-http-client-transport-9.4.24.v20191120.jar"/>
+ <library name="jcl-over-slf4j-1.7.24.jar"/>
+ <library name="slf4j-api-1.7.24.jar"/>
Review comment:
```
$> bin/nutch indexchecker -Dplugin.includes='protocol-okhttp|parse-tika|index-(basic|more|anchor)|indexer-solr' \
-doIndex https://localhost/
...
Exception in thread "main" java.lang.LinkageError: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of org/apache/nutch/plugin/PluginClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:418)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
at org.apache.nutch.indexwriter.solr.SolrIndexWriter.<clinit>(SolrIndexWriter.java:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:164)
at org.apache.nutch.indexer.IndexWriters.<init>(IndexWriters.java:97)
at org.apache.nutch.indexer.IndexWriters.lambda$get$0(IndexWriters.java:60)
at java.util.Map.computeIfAbsent(Map.java:957)
at org.apache.nutch.indexer.IndexWriters.get(IndexWriters.java:60)
at org.apache.nutch.indexer.IndexingFiltersChecker.process(IndexingFiltersChecker.java:295)
at org.apache.nutch.util.AbstractChecker.processSingle(AbstractChecker.java:85)
at org.apache.nutch.indexer.IndexingFiltersChecker.run(IndexingFiltersChecker.java:132)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.nutch.indexer.IndexingFiltersChecker.main(IndexingFiltersChecker.java:305)
```
Nutch already provides slf4j-api (version 1.7.26) which causes this error. When class implementations are shared between plugins and the Nutch core classes (obvious for logging classes because all write into the same log file) it's mandatory to add an exclusion.
I've successfully tried using the following exclusions:
```xml
<dependency org="org.apache.solr" name="solr-solrj" rev="8.5.1">
<!-- exclusions of dependencies provided in Nutch core (ivy/ivy.xml) -->
<exclude org="org.apache.commons" name="commons-codec" />
<exclude org="org.apache.commons" name="commons-logging" />
<exclude org="org.slf4j" name="slf4j-api" />
</dependency>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nutch] sebastian-nagel commented on pull request #515: NUTCH-2780 : Upgrade index-solr to use Solr 8.5.1
Posted by GitBox <gi...@apache.org>.
sebastian-nagel commented on pull request #515:
URL: https://github.com/apache/nutch/pull/515#issuecomment-620019631
Excellent! Indexing works now out of the box. I'll commit soon. Thanks, @balashashanka!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org