You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by ST ST <st...@gmail.com> on 2009/11/14 00:03:57 UTC
Fwd: Lucene MMAP Usage with Solr
Folks,
I am trying to get Lucene MMAP to work in solr.
I am assuming that when I configure MMAP the entire index will be loaded
into RAM.
Is that the right assumption ?
I have tried the following ways for using MMAP:
Option 1. Using the solr config below for MMAP configuration
-Dorg.apache.lucene.FSDirectory.class=org.apache.lucene.store.MMapDirectory
With this config, when I start solr with a 30G index, I expected that the
RAM usage should go up, but it did not.
Option 2. By Code Change
I made the following code change :
Changed org.apache.solr.core.StandardDirectoryFactory to use MMAP instead
of FSDirectory.
Code snippet pasted below.
Could you help me to understand if these are the right way to use MMAP?
Thanks much
/ST.
Code SNippet for Option 2:
package org.apache.solr.core;
/**
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import java.io.File;
import java.io.IOException;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.MMapDirectory;
/**
* Directory provider which mimics original Solr FSDirectory based behavior.
*
*/
public class StandardDirectoryFactory extends DirectoryFactory {
public Directory open(String path) throws IOException {
return MMapDirectory.open(new File(path));
}
}
Re: Fwd: Lucene MMAP Usage with Solr
Posted by Lance Norskog <go...@gmail.com>.
http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=/apis/mmap.htm
Normally file I/O in a program means that the data is copied between
the system I/O disk cache and the program's memory. Memory-mapping
means that the program address space points to the disk I/O cache
directly, there is no copying. In other words the program and the OS
share the same memory.
The OS tends to stream the entire file in but this is not required.
Memory-mapping may be faster and may not, depending on your index
sizes, memory access patterns, etc.
On Fri, Nov 13, 2009 at 7:49 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> I thought that was the way to use it (but I've never had to use it myself) and that it means memory through the roof, yes.
> If you look at the Solr Admin statistics page, does it show you which Directory you are using?
>
> For example, on 1 Solr instance I'm looking at I see:
>
> readerDir : org.apache.lucene.store.NIOFSDirectory@/mnt/............
>
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
>
> ----- Original Message ----
>> From: ST ST <st...@gmail.com>
>> To: solr-user@lucene.apache.org
>> Sent: Fri, November 13, 2009 6:03:57 PM
>> Subject: Fwd: Lucene MMAP Usage with Solr
>>
>> Folks,
>>
>> I am trying to get Lucene MMAP to work in solr.
>>
>> I am assuming that when I configure MMAP the entire index will be loaded
>> into RAM.
>> Is that the right assumption ?
>>
>> I have tried the following ways for using MMAP:
>>
>> Option 1. Using the solr config below for MMAP configuration
>>
>> -Dorg.apache.lucene.FSDirectory.class=org.apache.lucene.store.MMapDirectory
>>
>> With this config, when I start solr with a 30G index, I expected that the
>> RAM usage should go up, but it did not.
>>
>> Option 2. By Code Change
>> I made the following code change :
>>
>> Changed org.apache.solr.core.StandardDirectoryFactory to use MMAP instead
>> of FSDirectory.
>> Code snippet pasted below.
>>
>>
>> Could you help me to understand if these are the right way to use MMAP?
>>
>> Thanks much
>> /ST.
>>
>> Code SNippet for Option 2:
>>
>> package org.apache.solr.core;
>> /**
>> * Licensed to the Apache Software Foundation (ASF) under one or more
>> * contributor license agreements. See the NOTICE file distributed with
>> * this work for additional information regarding copyright ownership.
>> * The ASF licenses this file to You under the Apache License, Version 2.0
>> * (the "License"); you may not use this file except in compliance with
>> * the License. You may obtain a copy of the License at
>> *
>> * http://www.apache.org/licenses/LICENSE-2.0
>> *
>> * Unless required by applicable law or agreed to in writing, software
>> * distributed under the License is distributed on an "AS IS" BASIS,
>> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> * See the License for the specific language governing permissions and
>> * limitations under the License.
>> */
>>
>> import java.io.File;
>> import java.io.IOException;
>>
>> import org.apache.lucene.store.Directory;
>> import org.apache.lucene.store.MMapDirectory;
>>
>> /**
>> * Directory provider which mimics original Solr FSDirectory based behavior.
>> *
>> */
>> public class StandardDirectoryFactory extends DirectoryFactory {
>>
>> public Directory open(String path) throws IOException {
>> return MMapDirectory.open(new File(path));
>> }
>> }
>
>
--
Lance Norskog
goksron@gmail.com
Re: Fwd: Lucene MMAP Usage with Solr
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I thought that was the way to use it (but I've never had to use it myself) and that it means memory through the roof, yes.
If you look at the Solr Admin statistics page, does it show you which Directory you are using?
For example, on 1 Solr instance I'm looking at I see:
readerDir : org.apache.lucene.store.NIOFSDirectory@/mnt/............
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
----- Original Message ----
> From: ST ST <st...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Fri, November 13, 2009 6:03:57 PM
> Subject: Fwd: Lucene MMAP Usage with Solr
>
> Folks,
>
> I am trying to get Lucene MMAP to work in solr.
>
> I am assuming that when I configure MMAP the entire index will be loaded
> into RAM.
> Is that the right assumption ?
>
> I have tried the following ways for using MMAP:
>
> Option 1. Using the solr config below for MMAP configuration
>
> -Dorg.apache.lucene.FSDirectory.class=org.apache.lucene.store.MMapDirectory
>
> With this config, when I start solr with a 30G index, I expected that the
> RAM usage should go up, but it did not.
>
> Option 2. By Code Change
> I made the following code change :
>
> Changed org.apache.solr.core.StandardDirectoryFactory to use MMAP instead
> of FSDirectory.
> Code snippet pasted below.
>
>
> Could you help me to understand if these are the right way to use MMAP?
>
> Thanks much
> /ST.
>
> Code SNippet for Option 2:
>
> package org.apache.solr.core;
> /**
> * Licensed to the Apache Software Foundation (ASF) under one or more
> * contributor license agreements. See the NOTICE file distributed with
> * this work for additional information regarding copyright ownership.
> * The ASF licenses this file to You under the Apache License, Version 2.0
> * (the "License"); you may not use this file except in compliance with
> * the License. You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
>
> import java.io.File;
> import java.io.IOException;
>
> import org.apache.lucene.store.Directory;
> import org.apache.lucene.store.MMapDirectory;
>
> /**
> * Directory provider which mimics original Solr FSDirectory based behavior.
> *
> */
> public class StandardDirectoryFactory extends DirectoryFactory {
>
> public Directory open(String path) throws IOException {
> return MMapDirectory.open(new File(path));
> }
> }