You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by zhoucheng2008 <zh...@gmail.com> on 2013/01/25 15:30:25 UTC
回复： 回复： IndexReader.open and CorruptIndexException
SearcherManager mgr = new SearcherManager(dir, new MySearchWarmer());Can I put null as the second parameter in the SearcherManager constructor?If not, how to create a MySearchWarmer instance? The following explanation doesn't provide sufficient info:
class MySearchWarmer implements SearchWarmer { @Override public void warm(IndexSearcher searcher) throws IOException { // Run some diverse searches, searching and sorting against all // fields that are used by your application } }
------------------ 原始邮件 ------------------
发件人: "Michael McCandless"<lu...@mikemccandless.com>;
发送时间: 2013年1月25日(星期五) 晚上9:26
收件人: "java-user"<ja...@lucene.apache.org>;
主题: Re: 回复： IndexReader.open and CorruptIndexException
Maybe here?:
http://blog.mikemccandless.com/2011/11/near-real-time-readers-with-lucenes.html
Mike McCandless
http://blog.mikemccandless.com
On Fri, Jan 25, 2013 at 7:58 AM, Cheng <zh...@gmail.com> wrote:
> Any example code for this SearcherManager?
>
> On Fri, Jan 25, 2013 at 3:59 AM, Ian Lea <ia...@gmail.com> wrote:
>
>> There will be one file handle for every currently open file.
>>
>> Use SearcherManager and this problem should go away.
>>
>>
>> --
>> Ian.
>>
>>
>> On Thu, Jan 24, 2013 at 6:40 PM, zhoucheng2008 <zh...@gmail.com>
>> wrote:
>> > What file handlers did you guy refer to?
>> >
>> >
>> > I opened the index directory only. Is this the file handler? Also, how
>> to safely and effectively close the index directory?
>> >
>> >
>> > I found the link's explanation somewhat self-contradictory. After I read
>> it, I am confused if I should close the file handlers in the finally block
>> or not. I am using Java.
>> >
>> >
>> >
>> >
>> > ------------------ 原始邮件 ------------------
>> > 发件人: "Ian Lea"<ia...@gmail.com>;
>> > 发送时间: 2013年1月24日(星期四) 下午5:46
>> > 收件人: "java-user"<ja...@lucene.apache.org>;
>> >
>> > 主题: Re: IndexReader.open and CorruptIndexException
>> >
>> >
>> >
>> > Well, raising the limits is one option but there may be better ones.
>> >
>> > There's an FAQ entry on this:
>> >
>> http://wiki.apache.org/lucene-java/LuceneFAQ#Why_am_I_getting_an_IOException_that_says_.22Too_many_open_files.22.3F
>> >
>> > Take a look at org.apache.lucene.search.SearcherManager "Utility class
>> > to safely share IndexSearcher instances across multiple threads".
>> >
>> >
>> > --
>> > Ian.
>> >
>> >
>> > On Thu, Jan 24, 2013 at 9:18 AM, Rafał Kuć <r....@solr.pl> wrote:
>> >> Hello!
>> >>
>> >> You need to allow the user that is running Lucene to open more files.
>> >> There are plenty of tutorials available on the web. Modify your
>> >> /etc/security/limits.conf and if for example your user is lucene add
>> >> the following (or modify if those already exists):
>> >>
>> >> lucene soft nofile 64000
>> >> lucene hard nofile 64000
>> >>
>> >> Relog and run:
>> >>
>> >> sudo -u lucene -s "ulimit -Sn"
>> >>
>> >> To see if the limits are the ones you set. If they are not, check if
>> >> you don't have pam_limits.so commented in the /etc/pam.d/
>> >>
>> >> --
>> >> Regards,
>> >> Rafał Kuć
>> >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> >>
>> >>> Here is the log:
>> >>
>> >>> Jan 24, 2013 4:10:33 AM
>> >>> org.apache.tomcat.util.net.AprEndpoint$Acceptor run
>> >>> SEVERE: Socket accept failed
>> >>> org.apache.tomcat.jni.Error: 24: Too many open files
>> >>> at org.apache.tomcat.jni.Socket.accept(Native Method)
>> >>> at
>> >>>
>> org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:990)
>> >>> at java.lang.Thread.run(Thread.java:722)
>> >>
>> >>
>> >>
>> >>> Too many open files... How to solve it?
>> >>
>> >>
>> >>> On Tue, Jan 22, 2013 at 10:52 PM, Michael McCandless <
>> >>> lucene@mikemccandless.com> wrote:
>> >>
>> >>>> Can you post the full stack trace of the CorruptIndexException?
>> >>>>
>> >>>> Mike McCandless
>> >>>>
>> >>>> http://blog.mikemccandless.com
>> >>>>
>> >>>> On Tue, Jan 22, 2013 at 8:20 AM, Cheng <zh...@gmail.com>
>> wrote:
>> >>>> > Hi,
>> >>>> >
>> >>>> > I run a Lucene application on Tomcat. The app will try to open a
>> Linux
>> >>>> > directory, and sometime returns CorruptIndexException error.
>> >>>> >
>> >>>> > Shortly after I restart Tomcat (nothing else changes), the app can
>> be run
>> >>>> > on the fly. I am using the following statements to open a directory:
>> >>>> >
>> >>>> >
>> >>>> > try {
>> >>>> > searcher = new IndexSearcher(IndexReader.open(new
>> >>>> NIOFSDirectory(new
>> >>>> > File("/home/user/"+ type))));
>> >>>> > } catch (IOException e) {
>> >>>> > throw new Exception("[" + type + "] Cannot open index
>> folder...");
>> >>>> > }
>> >>>> >
>> >>>> > I would like to know how to tackle this problem.
>> >>>> >
>> >>>> > Many thanks!
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>>>
>> >>>>
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 回复： 回复： IndexReader.open and CorruptIndexException
Posted by Ian Lea <ia...@gmail.com>.
What version of lucene are you on? For 4.x and, as far as I can
recall, 3.x, I've just got
SearcherManager sm = new SearcherManager(dir, new SearcherFactory());
--
Ian.
On Fri, Jan 25, 2013 at 2:30 PM, zhoucheng2008 <zh...@gmail.com> wrote:
> SearcherManager mgr = new SearcherManager(dir, new MySearchWarmer());Can I put null as the second parameter in the SearcherManager constructor?If not, how to create a MySearchWarmer instance? The following explanation doesn't provide sufficient info:
> class MySearchWarmer implements SearchWarmer { @Override public void warm(IndexSearcher searcher) throws IOException { // Run some diverse searches, searching and sorting against all // fields that are used by your application } }
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Michael McCandless"<lu...@mikemccandless.com>;
> 发送时间: 2013年1月25日(星期五) 晚上9:26
> 收件人: "java-user"<ja...@lucene.apache.org>;
>
> 主题: Re: 回复： IndexReader.open and CorruptIndexException
>
>
>
> Maybe here?:
>
> http://blog.mikemccandless.com/2011/11/near-real-time-readers-with-lucenes.html
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Fri, Jan 25, 2013 at 7:58 AM, Cheng <zh...@gmail.com> wrote:
>> Any example code for this SearcherManager?
>>
>> On Fri, Jan 25, 2013 at 3:59 AM, Ian Lea <ia...@gmail.com> wrote:
>>
>>> There will be one file handle for every currently open file.
>>>
>>> Use SearcherManager and this problem should go away.
>>>
>>>
>>> --
>>> Ian.
>>>
>>>
>>> On Thu, Jan 24, 2013 at 6:40 PM, zhoucheng2008 <zh...@gmail.com>
>>> wrote:
>>> > What file handlers did you guy refer to?
>>> >
>>> >
>>> > I opened the index directory only. Is this the file handler? Also, how
>>> to safely and effectively close the index directory?
>>> >
>>> >
>>> > I found the link's explanation somewhat self-contradictory. After I read
>>> it, I am confused if I should close the file handlers in the finally block
>>> or not. I am using Java.
>>> >
>>> >
>>> >
>>> >
>>> > ------------------ 原始邮件 ------------------
>>> > 发件人: "Ian Lea"<ia...@gmail.com>;
>>> > 发送时间: 2013年1月24日(星期四) 下午5:46
>>> > 收件人: "java-user"<ja...@lucene.apache.org>;
>>> >
>>> > 主题: Re: IndexReader.open and CorruptIndexException
>>> >
>>> >
>>> >
>>> > Well, raising the limits is one option but there may be better ones.
>>> >
>>> > There's an FAQ entry on this:
>>> >
>>> http://wiki.apache.org/lucene-java/LuceneFAQ#Why_am_I_getting_an_IOException_that_says_.22Too_many_open_files.22.3F
>>> >
>>> > Take a look at org.apache.lucene.search.SearcherManager "Utility class
>>> > to safely share IndexSearcher instances across multiple threads".
>>> >
>>> >
>>> > --
>>> > Ian.
>>> >
>>> >
>>> > On Thu, Jan 24, 2013 at 9:18 AM, Rafał Kuć <r....@solr.pl> wrote:
>>> >> Hello!
>>> >>
>>> >> You need to allow the user that is running Lucene to open more files.
>>> >> There are plenty of tutorials available on the web. Modify your
>>> >> /etc/security/limits.conf and if for example your user is lucene add
>>> >> the following (or modify if those already exists):
>>> >>
>>> >> lucene soft nofile 64000
>>> >> lucene hard nofile 64000
>>> >>
>>> >> Relog and run:
>>> >>
>>> >> sudo -u lucene -s "ulimit -Sn"
>>> >>
>>> >> To see if the limits are the ones you set. If they are not, check if
>>> >> you don't have pam_limits.so commented in the /etc/pam.d/
>>> >>
>>> >> --
>>> >> Regards,
>>> >> Rafał Kuć
>>> >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>>> >>
>>> >>> Here is the log:
>>> >>
>>> >>> Jan 24, 2013 4:10:33 AM
>>> >>> org.apache.tomcat.util.net.AprEndpoint$Acceptor run
>>> >>> SEVERE: Socket accept failed
>>> >>> org.apache.tomcat.jni.Error: 24: Too many open files
>>> >>> at org.apache.tomcat.jni.Socket.accept(Native Method)
>>> >>> at
>>> >>>
>>> org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:990)
>>> >>> at java.lang.Thread.run(Thread.java:722)
>>> >>
>>> >>
>>> >>
>>> >>> Too many open files... How to solve it?
>>> >>
>>> >>
>>> >>> On Tue, Jan 22, 2013 at 10:52 PM, Michael McCandless <
>>> >>> lucene@mikemccandless.com> wrote:
>>> >>
>>> >>>> Can you post the full stack trace of the CorruptIndexException?
>>> >>>>
>>> >>>> Mike McCandless
>>> >>>>
>>> >>>> http://blog.mikemccandless.com
>>> >>>>
>>> >>>> On Tue, Jan 22, 2013 at 8:20 AM, Cheng <zh...@gmail.com>
>>> wrote:
>>> >>>> > Hi,
>>> >>>> >
>>> >>>> > I run a Lucene application on Tomcat. The app will try to open a
>>> Linux
>>> >>>> > directory, and sometime returns CorruptIndexException error.
>>> >>>> >
>>> >>>> > Shortly after I restart Tomcat (nothing else changes), the app can
>>> be run
>>> >>>> > on the fly. I am using the following statements to open a directory:
>>> >>>> >
>>> >>>> >
>>> >>>> > try {
>>> >>>> > searcher = new IndexSearcher(IndexReader.open(new
>>> >>>> NIOFSDirectory(new
>>> >>>> > File("/home/user/"+ type))));
>>> >>>> > } catch (IOException e) {
>>> >>>> > throw new Exception("[" + type + "] Cannot open index
>>> folder...");
>>> >>>> > }
>>> >>>> >
>>> >>>> > I would like to know how to tackle this problem.
>>> >>>> >
>>> >>>> > Many thanks!
>>> >>>>
>>> >>>> ---------------------------------------------------------------------
>>> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >>>>
>>> >>>>
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >>
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 回复： 回复： IndexReader.open and CorruptIndexException
Posted by Michael McCandless <lu...@mikemccandless.com>.
You can pass null for the SearcherFactory ... then SearcherManager
will just do new IndexSearcher(reader) for you.
Mike McCandless
http://blog.mikemccandless.com
On Fri, Jan 25, 2013 at 9:30 AM, zhoucheng2008 <zh...@gmail.com> wrote:
> SearcherManager mgr = new SearcherManager(dir, new MySearchWarmer());Can I put null as the second parameter in the SearcherManager constructor?If not, how to create a MySearchWarmer instance? The following explanation doesn't provide sufficient info:
> class MySearchWarmer implements SearchWarmer { @Override public void warm(IndexSearcher searcher) throws IOException { // Run some diverse searches, searching and sorting against all // fields that are used by your application } }
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Michael McCandless"<lu...@mikemccandless.com>;
> 发送时间: 2013年1月25日(星期五) 晚上9:26
> 收件人: "java-user"<ja...@lucene.apache.org>;
>
> 主题: Re: 回复： IndexReader.open and CorruptIndexException
>
>
>
> Maybe here?:
>
> http://blog.mikemccandless.com/2011/11/near-real-time-readers-with-lucenes.html
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Fri, Jan 25, 2013 at 7:58 AM, Cheng <zh...@gmail.com> wrote:
>> Any example code for this SearcherManager?
>>
>> On Fri, Jan 25, 2013 at 3:59 AM, Ian Lea <ia...@gmail.com> wrote:
>>
>>> There will be one file handle for every currently open file.
>>>
>>> Use SearcherManager and this problem should go away.
>>>
>>>
>>> --
>>> Ian.
>>>
>>>
>>> On Thu, Jan 24, 2013 at 6:40 PM, zhoucheng2008 <zh...@gmail.com>
>>> wrote:
>>> > What file handlers did you guy refer to?
>>> >
>>> >
>>> > I opened the index directory only. Is this the file handler? Also, how
>>> to safely and effectively close the index directory?
>>> >
>>> >
>>> > I found the link's explanation somewhat self-contradictory. After I read
>>> it, I am confused if I should close the file handlers in the finally block
>>> or not. I am using Java.
>>> >
>>> >
>>> >
>>> >
>>> > ------------------ 原始邮件 ------------------
>>> > 发件人: "Ian Lea"<ia...@gmail.com>;
>>> > 发送时间: 2013年1月24日(星期四) 下午5:46
>>> > 收件人: "java-user"<ja...@lucene.apache.org>;
>>> >
>>> > 主题: Re: IndexReader.open and CorruptIndexException
>>> >
>>> >
>>> >
>>> > Well, raising the limits is one option but there may be better ones.
>>> >
>>> > There's an FAQ entry on this:
>>> >
>>> http://wiki.apache.org/lucene-java/LuceneFAQ#Why_am_I_getting_an_IOException_that_says_.22Too_many_open_files.22.3F
>>> >
>>> > Take a look at org.apache.lucene.search.SearcherManager "Utility class
>>> > to safely share IndexSearcher instances across multiple threads".
>>> >
>>> >
>>> > --
>>> > Ian.
>>> >
>>> >
>>> > On Thu, Jan 24, 2013 at 9:18 AM, Rafał Kuć <r....@solr.pl> wrote:
>>> >> Hello!
>>> >>
>>> >> You need to allow the user that is running Lucene to open more files.
>>> >> There are plenty of tutorials available on the web. Modify your
>>> >> /etc/security/limits.conf and if for example your user is lucene add
>>> >> the following (or modify if those already exists):
>>> >>
>>> >> lucene soft nofile 64000
>>> >> lucene hard nofile 64000
>>> >>
>>> >> Relog and run:
>>> >>
>>> >> sudo -u lucene -s "ulimit -Sn"
>>> >>
>>> >> To see if the limits are the ones you set. If they are not, check if
>>> >> you don't have pam_limits.so commented in the /etc/pam.d/
>>> >>
>>> >> --
>>> >> Regards,
>>> >> Rafał Kuć
>>> >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>>> >>
>>> >>> Here is the log:
>>> >>
>>> >>> Jan 24, 2013 4:10:33 AM
>>> >>> org.apache.tomcat.util.net.AprEndpoint$Acceptor run
>>> >>> SEVERE: Socket accept failed
>>> >>> org.apache.tomcat.jni.Error: 24: Too many open files
>>> >>> at org.apache.tomcat.jni.Socket.accept(Native Method)
>>> >>> at
>>> >>>
>>> org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:990)
>>> >>> at java.lang.Thread.run(Thread.java:722)
>>> >>
>>> >>
>>> >>
>>> >>> Too many open files... How to solve it?
>>> >>
>>> >>
>>> >>> On Tue, Jan 22, 2013 at 10:52 PM, Michael McCandless <
>>> >>> lucene@mikemccandless.com> wrote:
>>> >>
>>> >>>> Can you post the full stack trace of the CorruptIndexException?
>>> >>>>
>>> >>>> Mike McCandless
>>> >>>>
>>> >>>> http://blog.mikemccandless.com
>>> >>>>
>>> >>>> On Tue, Jan 22, 2013 at 8:20 AM, Cheng <zh...@gmail.com>
>>> wrote:
>>> >>>> > Hi,
>>> >>>> >
>>> >>>> > I run a Lucene application on Tomcat. The app will try to open a
>>> Linux
>>> >>>> > directory, and sometime returns CorruptIndexException error.
>>> >>>> >
>>> >>>> > Shortly after I restart Tomcat (nothing else changes), the app can
>>> be run
>>> >>>> > on the fly. I am using the following statements to open a directory:
>>> >>>> >
>>> >>>> >
>>> >>>> > try {
>>> >>>> > searcher = new IndexSearcher(IndexReader.open(new
>>> >>>> NIOFSDirectory(new
>>> >>>> > File("/home/user/"+ type))));
>>> >>>> > } catch (IOException e) {
>>> >>>> > throw new Exception("[" + type + "] Cannot open index
>>> folder...");
>>> >>>> > }
>>> >>>> >
>>> >>>> > I would like to know how to tackle this problem.
>>> >>>> >
>>> >>>> > Many thanks!
>>> >>>>
>>> >>>> ---------------------------------------------------------------------
>>> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >>>>
>>> >>>>
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> >>
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org