You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Don Bosco Durai <bo...@apache.org> on 2016/06/20 17:09:51 UTC
Re: Encryption to Solr indexes – Using Custom Codec
Mohit, just curious. Have you considered just encrypting the volume where there index is stored? It will be done at the OS level and performance wise it will be much better and easier to manage. Would it address your requirement? Or is it purely academic exercise for you.
Bosco
From: "Sidana, Mohit" <Mo...@wolterskluwer.com>
Reply-To: <so...@lucene.apache.org>
Date: Monday, June 20, 2016 at 7:42 AM
To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
Subject: Encryption to Solr indexes – Using Custom Codec
Hello,
As Part of my studies I am exploring the solutions which can be used for Lucene/Solr Index encryption.
I found the patch open on Apache JIRA - Codec for index-level encryption (LUCENE-6966). https://issues.apache.org/jira/browse/LUCENE-6966 and I am currently trying to test this Custom codec with Solr to perform secure search over some sensitive records.
I've decided to follow the path described in Solr wiki, setting up Simple Text Codec and further tried to use Encrypted codec Source.
Here are the additional details.
I've created a basic jar file out of this source code (Build it as a jar from Eclipse using Maven Plugin).
The Solr installation I'm using to test this is the Solr 6.0.0 unzipped, and started via its embedded Jetty server and using the single core.
I've placed my jar with the codec in [My_Core\ instance Dir.]\ Lib
In:
[$SolrDir]\Solr\ My_Core \conf\solrconfig.xml
I've added the following lines:
<lib dir="./lib/" regex=".*\.jar"/>
<codecFactory class="solr.SchemaCodecFactory"/>
Then in the schema.xml file, I've declared some field and field Types that should use this codec:
<field name="City" type="strings"/>
<field name="Country" type="strings"/>
<field name="Name" type="strings"/>
<field name="Payment" type="strings"/>
<field name="Price" type="tlongs"/>
<field name="Product" type="strings"/>
<field name="State" type="strings"/>
<!-- set the per field posting format as below -->
<fieldType name="strings" class="solr.StrField" sortMissingLast="true" docValues="true" multiValued="true" postingsFormat="EncryptedLucene50"/>
<fieldType name="tlongs" class="solr.TrieLongField" positionIncrementGap="0" docValues="true" multiValued="true" precisionStep="8" postingsFormat="EncryptedLucene50"/>
I'm pretty sure I've followed all the steps described in Solr Wiki; however, when I actually try to use custom codec implementation (named "Encrypted Codec") to index some sample CSV data using simple post tool
java -Dtype=text/csv -Durl=http://localhost:8983/solr/My_Core /update -jar post.jar Sales.csv
and I have also tried doing the same with SolrJ but I have faced the same error.
SolrClient server = new HttpSolrClient("http://localhost:8983/solr/My_Core ");
SolrInputDocument doc = new SolrInputDocument();
doc.addField("id", "1234");
doc.addField("name", "A lovely summer holiday");
try {
server.add(doc);
server.commit();
System.out.println("Document added!");
} catch (SolrServerException | IOException e) {
e.printStackTrace();
}
}
}
I get the attached errors in Solr log.
org.apache.solr.common.SolrException: Exception writing document id b3e01ada-d0f1-4ddf-ad6a-2828bfe619a3 to the index; possible analysis error.
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:181)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:335)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:936)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1091)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:714)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:93)
at org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:391)
at org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:43)
at org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:360)
at org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:30)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:94)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2033)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1488)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:282)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:214)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169)
... 58 more
Caused by: java.lang.NoClassDefFoundError: org/apache/lucene/codecs/encrypted/DummyCipherFactory
at org.apache.lucene.codecs.encrypted.DummyEncryptedLucene60Codec$DummyEncryptedLucene50PostingsFormat.getCipherFactory(DummyEncryptedLucene60Codec.java:67)
at org.apache.lucene.codecs.encrypted.EncryptedLucene50PostingsFormat.fieldsConsumer(EncryptedLucene50PostingsFormat.java:479)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:196)
at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:107)
at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:134)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:423)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2815)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2989)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2956)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:619)
at org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1453)
at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1256)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.codecs.encrypted.DummyCipherFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 22 more
From the output I assume that Solr is not picking up the jar with this custom codec, and I don't get why?
What I find strange is that, I have also tried to run this codec with Lucene and eclipse by writing some code for indexing and searching and same codec jar is working just fine.
System.out.println("Indexing to directory '" + indexPath + "'...");
Directory dir = FSDirectory.open(Paths.get(indexPath));
Analyzer analyzer = new StandardAnalyzer();
IndexWriterConfig iwc = new IndexWriterConfig(analyzer);
// set encrypted codec //This encrypted Codec is being used for indexing
iwc.setCodec(new DummyEncryptedLucene60Codec());
if (create) {
// Create a new index in the directory, removing any
// previously indexed documents:
iwc.setOpenMode(OpenMode.CREATE);
} else {
// Add new documents to an existing index:
iwc.setOpenMode(OpenMode.CREATE_OR_APPEND);
}
IndexWriter writer = new IndexWriter(dir, iwc);
indexDocs(writer, docDir);
writer.close();
I'm guessing that As Solr uses Java SPI options to load this codec which loads things differently, and there's something missing...?
My question is, what might I be doing wrong here or what's missing so that codec jar can be picked up by Solr?
Thanks.
RE: Encryption to Solr indexes – Using Custom Codec
Posted by "Sidana, Mohit" <Mo...@wolterskluwer.com>.
Hello Bosco,
As part of my thesis I am evaluating the existing solutions for Index level Encryption.
I believe using this custom codec implementation would be a better choice as it provides very low level access and better control over index data.(As I am only intrested in how term dictionary and stored fields are being stored on disk)
Therefore, I am trying to test this approach with Solr.
Thanks,
Mohit
-----Original Message-----
From: Don Bosco Durai [mailto:bosco@apache.org]
Sent: Monday, June 20, 2016 7:10 PM
To: solr-user@lucene.apache.org
Subject: Re: Encryption to Solr indexes – Using Custom Codec
Mohit, just curious. Have you considered just encrypting the volume where there index is stored? It will be done at the OS level and performance wise it will be much better and easier to manage. Would it address your requirement? Or is it purely academic exercise for you.
Bosco
From: "Sidana, Mohit" <Mo...@wolterskluwer.com>
Reply-To: <so...@lucene.apache.org>
Date: Monday, June 20, 2016 at 7:42 AM
To: "solr-user@lucene.apache.org" <so...@lucene.apache.org>
Subject: Encryption to Solr indexes – Using Custom Codec
Hello,
As Part of my studies I am exploring the solutions which can be used for Lucene/Solr Index encryption.
I found the patch open on Apache JIRA - Codec for index-level encryption (LUCENE-6966). https://issues.apache.org/jira/browse/LUCENE-6966 and I am currently trying to test this Custom codec with Solr to perform secure search over some sensitive records.
I've decided to follow the path described in Solr wiki, setting up Simple Text Codec and further tried to use Encrypted codec Source.
Here are the additional details.
I've created a basic jar file out of this source code (Build it as a jar from Eclipse using Maven Plugin).
The Solr installation I'm using to test this is the Solr 6.0.0 unzipped, and started via its embedded Jetty server and using the single core.
I've placed my jar with the codec in [My_Core\ instance Dir.]\ Lib
In:
[$SolrDir]\Solr\ My_Core \conf\solrconfig.xml
I've added the following lines:
<lib dir="./lib/" regex=".*\.jar"/>
<codecFactory class="solr.SchemaCodecFactory"/>
Then in the schema.xml file, I've declared some field and field Types that should use this codec:
<field name="City" type="strings"/>
<field name="Country" type="strings"/>
<field name="Name" type="strings"/>
<field name="Payment" type="strings"/>
<field name="Price" type="tlongs"/>
<field name="Product" type="strings"/>
<field name="State" type="strings"/>
<!-- set the per field posting format as below -->
<fieldType name="strings" class="solr.StrField" sortMissingLast="true" docValues="true" multiValued="true" postingsFormat="EncryptedLucene50"/>
<fieldType name="tlongs" class="solr.TrieLongField" positionIncrementGap="0" docValues="true" multiValued="true" precisionStep="8" postingsFormat="EncryptedLucene50"/>
I'm pretty sure I've followed all the steps described in Solr Wiki; however, when I actually try to use custom codec implementation (named "Encrypted Codec") to index some sample CSV data using simple post tool
java -Dtype=text/csv -Durl=http://localhost:8983/solr/My_Core /update -jar post.jar Sales.csv
and I have also tried doing the same with SolrJ but I have faced the same error.
SolrClient server = new HttpSolrClient("http://localhost:8983/solr/My_Core ");
SolrInputDocument doc = new SolrInputDocument();
doc.addField("id", "1234");
doc.addField("name", "A lovely summer holiday");
try {
server.add(doc);
server.commit();
System.out.println("Document added!");
} catch (SolrServerException | IOException e) {
e.printStackTrace();
}
}
}
I get the attached errors in Solr log.
org.apache.solr.common.SolrException: Exception writing document id b3e01ada-d0f1-4ddf-ad6a-2828bfe619a3 to the index; possible analysis error.
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:181)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:335)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:936)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1091)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:714)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:93)
at org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:391)
at org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:43)
at org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:360)
at org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:30)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:94)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2033)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1488)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:282)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:214)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169)
... 58 more
Caused by: java.lang.NoClassDefFoundError: org/apache/lucene/codecs/encrypted/DummyCipherFactory
at org.apache.lucene.codecs.encrypted.DummyEncryptedLucene60Codec$DummyEncryptedLucene50PostingsFormat.getCipherFactory(DummyEncryptedLucene60Codec.java:67)
at org.apache.lucene.codecs.encrypted.EncryptedLucene50PostingsFormat.fieldsConsumer(EncryptedLucene50PostingsFormat.java:479)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:196)
at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:107)
at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:134)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:423)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2815)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2989)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2956)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:619)
at org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1453)
at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1256)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.codecs.encrypted.DummyCipherFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 22 more
From the output I assume that Solr is not picking up the jar with this custom codec, and I don't get why?
What I find strange is that, I have also tried to run this codec with Lucene and eclipse by writing some code for indexing and searching and same codec jar is working just fine.
System.out.println("Indexing to directory '" + indexPath + "'...");
Directory dir = FSDirectory.open(Paths.get(indexPath));
Analyzer analyzer = new StandardAnalyzer();
IndexWriterConfig iwc = new IndexWriterConfig(analyzer);
// set encrypted codec //This encrypted Codec is being used for indexing
iwc.setCodec(new DummyEncryptedLucene60Codec());
if (create) {
// Create a new index in the directory, removing any
// previously indexed documents:
iwc.setOpenMode(OpenMode.CREATE);
} else {
// Add new documents to an existing index:
iwc.setOpenMode(OpenMode.CREATE_OR_APPEND);
}
IndexWriter writer = new IndexWriter(dir, iwc);
indexDocs(writer, docDir);
writer.close();
I'm guessing that As Solr uses Java SPI options to load this codec which loads things differently, and there's something missing...?
My question is, what might I be doing wrong here or what's missing so that codec jar can be picked up by Solr?
Thanks.