You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Maxence SAUNIER (JIRA)" <ji...@apache.org> on 2018/06/06 09:18:00 UTC
[jira] [Comment Edited] (SOLR-12283) Unable To Load
ZKPropertiesWriter when dih.jar is added as runtimelib BLOB in .system
collection
[ https://issues.apache.org/jira/browse/SOLR-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502952#comment-16502952 ]
Maxence SAUNIER edited comment on SOLR-12283 at 6/6/18 9:17 AM:
----------------------------------------------------------------
Hello Shawn,
In cloud? With Solr 7.3?
My process to add my DIH in Solr 7.3 inCloud:
1. Create .system for the BLOB
{code:java}
curl "http://localhost:8983/solr/admin/collections?action=CREATE&name=.system"{code}
2. Send Data Import Handler Jar on the BLOB
{code:java}
curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @solr-dataimporthandler-7.3.0-SNAPSHOT.jar http://localhost:8983/solr/.system/blob/data-import-handler{code}
3. Add on the runtime lib
{code:java}
curl http://localhost:8983/solr/unicorn/config -H 'Content-type:application/json' -d '{"add-runtimelib": { "name":"data-import-handler", "version":1 }}'{code}
4. Add MySQL connector Jar on the BLOB
{code:java}
curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @mysql-connector-java-5.1.46.jar http://localhost:8983/solr/.system/blob/mysql-connector-java{code}
5. Add MySQL Connector on the runtime lib
{code:java}
curl http://localhost:8983/solr/unicorn/config -H 'Content-type:application/json' -d '{"add-runtimelib": { "name":"mysql-connector-java", "version":1 }}'{code}
6. Create DIH Path on the cloud (zkcli.py if a script to overload default zkcli. It work good.)
{code:java}
./zkcli.py makepath /configs/unicorn/DIH{code}
7. Send to DIH folder the XML definition
{code:java}
./zkcli.py putfile /configs/unicorn/DIH/ DIH/test-dih.xml{code}
test-dih.xml =
{code:xml}
<dataConfig>
<dataSource name="Gesloc" type="JdbcDataSource" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://srv-gesloc-sql/TRANSACTIONCITYANEWLOCATION" user="secret" password="secret" />
<document>
<entity name="events_Gesloc" pk="id" dataSource="Gesloc" query="SELECT id from view_indexation_advertisements" >
<field column="id" name="id"/>
</entity>
</document>
</dataConfig>
{code}
8. Add request handler
{code}curl -X POST -H 'Content-type:application/json' -d @request_dih.json http://localhost:8983/solr/unicorn/config{code}
request_dih.json=
{code:json}
{
"add-requesthandler":
{
"name": "/test-dih",
"class": "org.apache.solr.handler.dataimport.DataImportHandler",
"runtimeLib": true,
"version": 1,
"defaults":
{
"config": "DIH/test-dih.xml"
}
}
}
{code}
9. Reload config
{code}curl http://localhost:8983/solr/unicorn/test-dih?command=reload-config&commit=true&debug=true{code}
Thanks for your help.
was (Author: moltroon):
Hello Shawn,
In cloud? With Solr 7.3?
My process to add my DIH in Solr 7.3 inCloud:
1. Create .system for the BLOB
{code:java}
curl "http://localhost:8983/solr/admin/collections?action=CREATE&name=.system"{code}
2. Send Data Import Handler Jar on the BLOB
{code:java}
curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @solr-dataimporthandler-7.3.0-SNAPSHOT.jar http://localhost:8983/solr/.system/blob/data-import-handler{code}
3. Add on the runtime lib
{code:java}
curl http://localhost:8983/solr/unicorn/config -H 'Content-type:application/json' -d '{"add-runtimelib": { "name":"data-import-handler", "version":1 }}'{code}
4. Add MySQL connector Jar on the BLOB
{code:java}
curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @mysql-connector-java-5.1.46.jar http://localhost:8983/solr/.system/blob/mysql-connector-java{code}
5. Add MySQL Connector on the runtime lib
{code:java}
curl http://localhost:8983/solr/unicorn/config -H 'Content-type:application/json' -d '{"add-runtimelib": { "name":"mysql-connector-java", "version":1 }}'{code}
6. Create DIH Path on the cloud (zkcli.py if a script to overload default zkcli. It work good.)
{code:java}
./zkcli.py makepath /configs/unicorn/DIH{code}
7. Send to DIH folder the XML definition
{code:java}
./zkcli.py putfile /configs/unicorn/DIH/ DIH/test-dih.xml{code}
test-dih.xml =
{code:xml}
<dataConfig>
<dataSource name="Gesloc" type="JdbcDataSource" driver="com.mysql.jdbc.Driver" url="jdbc:mysql://srv-gesloc-sql/TRANSACTIONCITYANEWLOCATION" user="secret" password="secret" />
<document>
<entity name="events_Gesloc" pk="id" dataSource="Gesloc" query="SELECT id from view_indexation_advertisements" >
<field column="id" name="id"/>
</entity>
</document>
</dataConfig>
{code}
Thanks for your help.
> Unable To Load ZKPropertiesWriter when dih.jar is added as runtimelib BLOB in .system collection
> ------------------------------------------------------------------------------------------------
>
> Key: SOLR-12283
> URL: https://issues.apache.org/jira/browse/SOLR-12283
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - DataImportHandler
> Affects Versions: 6.6.1, 7.3
> Environment: Debian
> SolrCloud
> Reporter: Maxence SAUNIER
> Priority: Blocker
> Attachments: modified-DIH.zip, modified-DIH.zip, mysql-connector-java-5.1.46-bin.jar, mysql-connector-java-5.1.46.jar, request_handler_config.json, solr-core-7.3.0.jar, solr-dataimporthandler-7.3.0.jar, solr-dataimporthandler-extras-7.3.0.jar, solr-solrj-7.3.0.jar, solr.log, solr.log, solr.log, solr.log
>
>
> Hello,
> It's been 2 weeks that I try to correct this problem with the community user-solr but no success. I seriously wonder if this is not a problem in the code. I do not have the impression that many people use DIH with Solr's cloud version.
> On Internet, no similar problem.
> For information, the following configuration of DIH comes from DIHs that work in production on a single Solr server. The connections to the databases are therefore correct.
> *Errors messages:*
> {panel:title=DataImporter}
> {code:java}
> Full Import failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to PropertyWriter implementation:ZKPropertiesWriter
> at org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:339)
> at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:420)
> at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
> at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:530)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at org.apache.solr.handler.dataimport.DocBuilder.loadClass(DocBuilder.java:935)
> at org.apache.solr.handler.dataimport.DataImporter.createPropertyWriter(DataImporter.java:335)
> ... 42 more
> {code}
> {panel}
> {panel:title=RequestHandlerBase}
> {code:java}
> java.lang.NullPointerException
> at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:429)
> at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
> at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:530)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> {panel}
> {panel:title=HttpSolrCall}
> {code:java}
> null:java.lang.NullPointerException
> at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:429)
> at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
> at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:183)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:517)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:384)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:330)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:530)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> {panel}
> *Process to reproduce the problem:*
> Create .system collection for the BLOB
> {code}
> curl http://localhost:8983/solr/admin/collections?action=CREATE&name=.system
> {code}
> Add Data Import Handler JAR file.
> {code}
> curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @solr-dataimporthandler-7.3.0.jar http://localhost:8983/solr/.system/blob/data-import-handler
> {code}
> Add the runtime lib on the configoverlay.json with the API
> {code}
> curl http://localhost:8983/solr/unicorn/config -H 'Content-type:application/json' -d '{"add-runtimelib": { "name":"data-import-handler", "version":1 }}'
> {code}
> Send mysql connector on the blob. I have test with the connector-bin.jar but I have the same error.
> {code}
> curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @mysql-connector-java-5.1.46.jar http://localhost:8983/solr/.system/blob/mysql-connector-java
> {code}
> Add runtime
> {code}
> curl http://localhost:8983/solr/unicorn/config -H 'Content-type:application/json' -d '{"add-runtimelib": { "name":"mysql-connector-java", "version":1 }}'
> {code}
> Personally, I create a DIH folder on the collection config path on the cloud. It's not necessary.
> And I put the DIH config file on the cloud with the zkcli (it's a personnal script to overload the zkcli.sh on Solr but it's the same result. My file is on the cloud.)
> {code}
> ./zkcli.py putfile /configs/unicorn/DIH/ DIH/indexation_events.xml
> {code}
> Add request handler config:
> {code}
> curl -X POST http://{solr_server}:8983/solr/{collection_name}/config -H 'Content-type:application/json' -d '
> {
> "add-requesthandler":
> {
> "name": "/test-dih",
> "class": "org.apache.solr.handler.dataimport.DataImportHandler",
> "runtimeLib": true,
> "version": 1,
> "defaults":
> {
> "config": "DIH/indexation_events.xml"
> }
> }
> } '
> {code}
> Reload the config:
> {code}
> curl http://localhost:8983/solr/unicorn/test-dih?command=reload-config&commit=true&debug=true
> {code}
> Import and have error:
> {code}
> curl http://localhost:8983/solr/unicorn/test-dih?command=full-import&commit=true&debug=true
> {code}
> I have test to add on request handler and BLOB this jar files but not change:
> # data-import-handler-extras
> # solr-core
> # solr-solrj
> I have join all files used in this process.
> Thansk you.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org