Using RAMDirectoryFactory in Master/Slave setup

[Nipun Bhatia <ni...@kosmix.com>]

Overview ­ Trying to evaluate if keeping the index in memory using
RAMDirectoryFactory can help in query performance.I am trying to perform the
indexing on the master using solr.StandardDirectoryFactory and on the slave
using solr.RAMDirectoryFactory

Details:
We have set-up Solr in a master/slave enviornment. The index is built on the
master and then replicated to slaves which are used to serve the query.
The replication is done using the in-built Java replication in Solr.
On the master, in the <indexDefaults> of solrconfig.xml we have
<directoryFactory name="DirectoryFactory"
        class="solr.StandardDirectoryFactory"/>

On the slave, I tried to use the following in the <indexDefaults>

<directoryFactory name="DirectoryFactory"
         class="solr.RAMDirectoryFactory"/>

My slave shows no data for any queries. In solrconfig.xml it is mentioned
that replication doesn¹t work when using RAMDirectoryFactory, however this (
https://issues.apache.org/jira/browse/SOLR-1379) mentions that you can use
it to have the index on disk and then load into memory.

To test the sanity of my set-up, I changed solrconfig.xml in the slave to
and replicated:
<directoryFactory name="DirectoryFactory"
        class="solr.StandardDirectoryFactory"/>
I was able to see the results.

Shouldn¹t RAMDirectoryFactory be used for reading index from disk into
memory? 

Any help/pointers in the right direction would be appreciated.

Thanks!

Re: Using RAMDirectoryFactory in Master/Slave setup

[nipunb <ni...@walmartlabs.com>]

Thanks for the pointer to MMapDirectoryFactory. 
Not having replication with RAMDirectoryFactory is a deal killer. We dont
want to index on the machines that serve queries.
>From what I can gather from reading, MMapDirectory + SSD could be a happy
medium. 
I'll try to evaluate these a bit more formally and post to the list.
Thanks for the help again!

--
View this message in context: http://lucene.472066.n3.nabble.com/Using-RAMDirectoryFactory-in-Master-Slave-setup-tp3112007p3114809.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Using RAMDirectoryFactory in Master/Slave setup

[eks dev <ek...@googlemail.com>]

Your best bet is MMapDirectoryFactory, you can come very close to the
performance of the RAMDirectory. Unfortunatelly this setup with
Master_on_disk->Slaves_in_ram type of setup is not possible using
solr.

We are moving our architecture to solr at the moment, and this is one
of "missings" we have to somehow figure out.

The problem is that MMap works fine on average, but it also has quirks
regarding upper quntiles of the responses.

If you are using RAMDirectory, you do not need to be afraid that
occasionally slow IO will kill performance for some of your requests.
This happens with MMAP, and not all that rare, depending on your usage
pattern (high update/commit rate for example). I repeat, RAMDirectory
is not to beat when it comes to reduction of the IO-caused "outliers".

We removed some 90% of the slowest response times by using
RAMDirectory instead of MMap...
Depending on what you want to optimize, MMap can work just fine for
you, and has some nice properties, eg.  you do not need to tune gc()
as much as if you manage bigger heap (RAMDirectory...)

But, imo, it would make sense to have some possibility to do it in solr.







On Mon, Jun 27, 2011 at 10:50 AM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> On Mon, Jun 27, 2011 at 12:49 PM, nipunb <ni...@walmartlabs.com> wrote:
>> I found a similar post -
>> http://lucene.472066.n3.nabble.com/Problems-with-RAMDirectory-in-Solr-td1575223.html
>> It mentions that Java based replication might work (This is what I have
>> used, but didn't work for me)
>
> Solr Replication does not work with non-file directory implementations.
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Using RAMDirectoryFactory in Master/Slave setup

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Mon, Jun 27, 2011 at 12:49 PM, nipunb <ni...@walmartlabs.com> wrote:
> I found a similar post -
> http://lucene.472066.n3.nabble.com/Problems-with-RAMDirectory-in-Solr-td1575223.html
> It mentions that Java based replication might work (This is what I have
> used, but didn't work for me)

Solr Replication does not work with non-file directory implementations.

-- 
Regards,
Shalin Shekhar Mangar.

Re: Using RAMDirectoryFactory in Master/Slave setup

[nipunb <ni...@walmartlabs.com>]

I found a similar post - 
http://lucene.472066.n3.nabble.com/Problems-with-RAMDirectory-in-Solr-td1575223.html
It mentions that Java based replication might work (This is what I have
used, but didn't work for me)
More interestingly it points out that OS's file system cache maybe able to
do this job better. 
Has anybody had a chance to do a comparison based on query performance for
StandardDirectoryFactory vs RAMDirectoryFactory.

--
View this message in context: http://lucene.472066.n3.nabble.com/Using-RAMDirectoryFactory-in-Master-Slave-setup-tp3112007p3112818.html
Sent from the Solr - User mailing list archive at Nabble.com.