You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by deniz <de...@gmail.com> on 2012/11/02 09:44:57 UTC
Solr Replication is not Possible on RAMDirectory?
Hi all, I am trying to set up a master/slave system, by following this page :
http://wiki.apache.org/solr/SolrReplication
I was able to set up and did some experiments with that, but when i try to
set the index for RAMDirectory, i got errors for indexing.
While master and slave are both using a non-RAM directory, everything is
okay... but when i try to use RAMdirectory on both I got this error below:
16:40:31.626 [qtp28208563-24] ERROR org.apache.solr.core.SolrCore -
org.apache.lucene.index.IndexNotFoundException: no segments* file found in
org.apache.lucene.store.RAMDirectory@7e693f
lockFactory=org.apache.lucene.store.NativeFSLockFactory@92c787: files: []
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:741)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:630)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:343)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:639)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:75)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:62)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:191)
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:77)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:511)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:87)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1016)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:351)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Unknown Source)
16:40:31.627 [qtp28208563-24] ERROR o.a.solr.servlet.SolrDispatchFilter -
null:org.apache.lucene.index.IndexNotFoundException: no segments* file found
in org.apache.lucene.store.RAMDirectory@7e693f
lockFactory=org.apache.lucene.store.NativeFSLockFactory@92c787: files: []
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:741)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:630)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:343)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:639)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:75)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:62)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:191)
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:77)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:511)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:87)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1016)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:351)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Unknown Source)
so it is not possible to use RAMdirectory for replication?
-----
Zeki ama calismiyor... Calissa yapar...
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication is not Possible on RAMDirectory?
Posted by Shawn Heisey <so...@elyograg.org>.
On 11/4/2012 11:41 PM, deniz wrote:
> Michael Della Bitta-2 wrote
>> No, RAMDirectory doesn't work for replication. Use MMapDirectory... it
>> ends up storing the index in RAM and more efficiently so, plus it's
>> backed by disk.
>>
>> Just be sure to not set a big heap because MMapDirectory works outside of
>> heap.
> for my tests, i dont think index is ended up in ram with mmap... i gave
> 4gigs for heap while using mmap and got mapping error while indexing...
> while index should be something around 2 gigs, ram consumption was around
> 300mbs...
With mmap, the ram is not actually "consumed" by your application, which
in this case is Java. The operating system is handling it --
transparently mapping the files on disk to a virtual memory space and
using excess RAM to cache that data and make it fast. If you have
enough extra memory (disk cache) to fit the entire index, the OS will
never have to read any part of the index from disk more than once. With
RAMDirectory, the index has to go into the Java heap, which is much less
efficient at memory management than the native operating system.
> Can anyone explain why RAMDirectory cant be used for replication? I cant see
> why the master is set for using RAMDirectory and replica is using MMap or
> some other? As far as I understand SolrCloud is some kinda pushing from
> master to replica/slave... so why it is not possible to push from RAM to
> HDD? If my logic is wrong, someone can please explain me all these?
With RAMDirectory, there are no files to copy. Replication does not
copy Solr (Lucene) documents, it copies files.
Thanks,
Shawn
Re: Solr Replication is not Possible on RAMDirectory?
Posted by Shawn Heisey <so...@elyograg.org>.
> Shawn Heisey-4 wrote
>> ... transparently mapping the files on disk to a virtual memory space
>> and
>> using excess RAM to cache that data and make it fast. If you have
>> enough extra memory (disk cache) to fit the entire index, the OS will
>> never have to read any part of the index from disk more than once....
>
> so for disk cache, are there any disks with 1 gigs or more of caches? if
> am
> not wrong there are mostly 16 or 32mb cache disks around (or i am checking
> the wrong stuff? ) if so, that amount definitely too small...
I am not talking about the cache on the actual disk drive, or even cache
on your hard drive controller. I am talking about the operating system
using RAM, specifically RAM not being used by programs, to cache data on
your hard drive. All modern operating systems do it, even the one made in
Redmond that people love to hate.
If you have 16 GB of RAM and all your programs use up 4.5 GB, you can
count on the OS using at least another half GB, so you have about 11 GB
left. The OS is going to put data that it reads and writes to/from your
disk in this space. If you start up another program that wants 2GB, the OS
will simply throw away 2 GB of data in its cache (it's still on the disk,
after all) and give that RAM to the new program.
Solr counts on this OS capability for good performance.
Thanks,
Shawn
Re: Solr Replication is not Possible on RAMDirectory?
Posted by Michael Della Bitta <mi...@appinions.com>.
Here's some reading:
http://en.wikipedia.org/wiki/Page_cache
Michael Della Bitta
------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
www.appinions.com
Where Influence Isn’t a Game
On Mon, Nov 5, 2012 at 8:02 PM, deniz <de...@gmail.com> wrote:
> Erik Hatcher-4 wrote
>> There's an open issue (with a patch!) that enables this, it seems:
>> <https://issues.apache.org/jira/browse/SOLR-3911>
>
> i will check it for sure, thank you Erik :)
>
>
> Shawn Heisey-4 wrote
>> ... transparently mapping the files on disk to a virtual memory space and
>> using excess RAM to cache that data and make it fast. If you have
>> enough extra memory (disk cache) to fit the entire index, the OS will
>> never have to read any part of the index from disk more than once....
>
> so for disk cache, are there any disks with 1 gigs or more of caches? if am
> not wrong there are mostly 16 or 32mb cache disks around (or i am checking
> the wrong stuff? ) if so, that amount definitely too small...
>
>
>
>
>
> -----
> Zeki ama calismiyor... Calissa yapar...
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018396.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication is not Possible on RAMDirectory?
Posted by deniz <de...@gmail.com>.
Erik Hatcher-4 wrote
> There's an open issue (with a patch!) that enables this, it seems:
> <https://issues.apache.org/jira/browse/SOLR-3911>
i will check it for sure, thank you Erik :)
Shawn Heisey-4 wrote
> ... transparently mapping the files on disk to a virtual memory space and
> using excess RAM to cache that data and make it fast. If you have
> enough extra memory (disk cache) to fit the entire index, the OS will
> never have to read any part of the index from disk more than once....
so for disk cache, are there any disks with 1 gigs or more of caches? if am
not wrong there are mostly 16 or 32mb cache disks around (or i am checking
the wrong stuff? ) if so, that amount definitely too small...
-----
Zeki ama calismiyor... Calissa yapar...
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018396.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication is not Possible on RAMDirectory?
Posted by deniz <de...@gmail.com>.
Erik Hatcher-4 wrote
> There's an open issue (with a patch!) that enables this, it seems:
> <https://issues.apache.org/jira/browse/SOLR-3911>
>
> Erik
well patch seems not doing that... i have tried and still getting some error
lines about the dir types
-----
Zeki ama calismiyor... Calissa yapar...
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018670.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication is not Possible on RAMDirectory?
Posted by Erik Hatcher <er...@gmail.com>.
There's an open issue (with a patch!) that enables this, it seems: <https://issues.apache.org/jira/browse/SOLR-3911>
Erik
On Nov 5, 2012, at 07:41 , deniz wrote:
> Michael Della Bitta-2 wrote
>> No, RAMDirectory doesn't work for replication. Use MMapDirectory... it
>> ends up storing the index in RAM and more efficiently so, plus it's
>> backed by disk.
>>
>> Just be sure to not set a big heap because MMapDirectory works outside of
>> heap.
>
> for my tests, i dont think index is ended up in ram with mmap... i gave
> 4gigs for heap while using mmap and got mapping error while indexing...
> while index should be something around 2 gigs, ram consumption was around
> 300mbs...
>
> Can anyone explain why RAMDirectory cant be used for replication? I cant see
> why the master is set for using RAMDirectory and replica is using MMap or
> some other? As far as I understand SolrCloud is some kinda pushing from
> master to replica/slave... so why it is not possible to push from RAM to
> HDD? If my logic is wrong, someone can please explain me all these?
>
>
>
> -----
> Zeki ama calismiyor... Calissa yapar...
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018198.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication is not Possible on RAMDirectory?
Posted by deniz <de...@gmail.com>.
Michael Della Bitta-2 wrote
> No, RAMDirectory doesn't work for replication. Use MMapDirectory... it
> ends up storing the index in RAM and more efficiently so, plus it's
> backed by disk.
>
> Just be sure to not set a big heap because MMapDirectory works outside of
> heap.
for my tests, i dont think index is ended up in ram with mmap... i gave
4gigs for heap while using mmap and got mapping error while indexing...
while index should be something around 2 gigs, ram consumption was around
300mbs...
Can anyone explain why RAMDirectory cant be used for replication? I cant see
why the master is set for using RAMDirectory and replica is using MMap or
some other? As far as I understand SolrCloud is some kinda pushing from
master to replica/slave... so why it is not possible to push from RAM to
HDD? If my logic is wrong, someone can please explain me all these?
-----
Zeki ama calismiyor... Calissa yapar...
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018198.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication is not Possible on RAMDirectory?
Posted by Michael Della Bitta <mi...@appinions.com>.
> so it is not possible to use RAMdirectory for replication?
No, RAMDirectory doesn't work for replication. Use MMapDirectory... it
ends up storing the index in RAM and more efficiently so, plus it's
backed by disk.
Just be sure to not set a big heap because MMapDirectory works outside of heap.
Michael Della Bitta
------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
www.appinions.com
Where Influence Isn’t a Game
On Fri, Nov 2, 2012 at 4:44 AM, deniz <de...@gmail.com> wrote:
> so it is not possible to use RAMdirectory for replication?