You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by zhoucheng2008 <zh...@gmail.com> on 2013/01/25 18:18:58 UTC
回复： 回复： 回复： 回复： 回复： IndexReader.open and CorruptIndexException
SearcherManager sm = new SearcherManager(dir, new SearcherFactory());
new SearchFactory() is not recognized in my lucene. asked to change to SearcherWarmer. Why?
I have 3.5
------------------ 原始邮件 ------------------
发件人: "Michael McCandless"<lu...@mikemccandless.com>;
发送时间: 2013年1月26日(星期六) 凌晨0:35
收件人: "java-user"<ja...@lucene.apache.org>;
主题: Re: 回复： 回复： 回复： 回复： IndexReader.open and CorruptIndexException
That should work.
Mike McCandless
http://blog.mikemccandless.com
On Fri, Jan 25, 2013 at 11:27 AM, zhoucheng2008 <zh...@gmail.com> wrote:
> Sorry, I meant this:
>
>
> SearcherManager sm = new SearcherManager(dir, new SearcherFactory());
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Ian Lea"<ia...@gmail.com>;
> 发送时间: 2013年1月26日(星期六) 凌晨0:16
> 收件人: "java-user"<ja...@lucene.apache.org>;
>
> 主题: Re: 回复： 回复： 回复： IndexReader.open and CorruptIndexException
>
>
>
>> Is SearcherFactory the same as SearcherManager?
>
> No.
>
>> Ian mentioned a new warmer() solution. Maybe I can try that first.
>
> Really? Don't remember that.
>
>
> --
> Ian.
>
>> ------------------ 原始邮件 ------------------
>> 发件人: "Michael McCandless"<lu...@mikemccandless.com>;
>> 发送时间: 2013年1月26日(星期六) 凌晨0:10
>> 收件人: "java-user"<ja...@lucene.apache.org>;
>>
>> 主题: Re: 回复： 回复： IndexReader.open and CorruptIndexException
>>
>>
>>
>> You can pass null for the SearcherFactory ... then SearcherManager
>> will just do new IndexSearcher(reader) for you.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Fri, Jan 25, 2013 at 9:30 AM, zhoucheng2008 <zh...@gmail.com> wrote:
>>> SearcherManager mgr = new SearcherManager(dir, new MySearchWarmer());Can I put null as the second parameter in the SearcherManager constructor?If not, how to create a MySearchWarmer instance? The following explanation doesn't provide sufficient info:
>>> class MySearchWarmer implements SearchWarmer { @Override public void warm(IndexSearcher searcher) throws IOException { // Run some diverse searches, searching and sorting against all // fields that are used by your application } }
>>>
>>>
>>>
>>>
>>>
>>>
>>> ------------------ 原始邮件 ------------------
>>> 发件人: "Michael McCandless"<lu...@mikemccandless..com>;
>>> 发送时间: 2013年1月25日(星期五) 晚上9:26
>>> 收件人: "java-user"<ja...@lucene.apache.org>;
>>>
>>> 主题: Re: 回复： IndexReader.open and CorruptIndexException
>>>
>>>
>>>
>>> Maybe here?:
>>>
>>> http://blog.mikemccandless.com/2011/11/near-real-time-readers-with-lucenes.html
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>> On Fri, Jan 25, 2013 at 7:58 AM, Cheng <zh...@gmail.com> wrote:
>>>> Any example code for this SearcherManager?
>>>>
>>>> On Fri, Jan 25, 2013 at 3:59 AM, Ian Lea <ia...@gmail.com> wrote:
>>>>
>>>>> There will be one file handle for every currently open file.
>>>>>
>>>>> Use SearcherManager and this problem should go away.
>>>>>
>>>>>
>>>>> --
>>>>> Ian.
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 6:40 PM, zhoucheng2008 <zh...@gmail.com>
>>>>> wrote:
>>>>> > What file handlers did you guy refer to?
>>>>> >
>>>>> >
>>>>> > I opened the index directory only. Is this the file handler? Also, how
>>>>> to safely and effectively close the index directory?
>>>>> >
>>>>> >
>>>>> > I found the link's explanation somewhat self-contradictory. After I read
>>>>> it, I am confused if I should close the file handlers in the finally block
>>>>> or not. I am using Java.
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > ------------------ 原始邮件 ------------------
>>>>> > 发件人: "Ian Lea"<ia...@gmail.com>;
>>>>> > 发送时间: 2013年1月24日(星期四) 下午5:46
>>>>> > 收件人: "java-user"<ja...@lucene.apache.org>;
>>>>> >
>>>>> > 主题: Re: IndexReader.open and CorruptIndexException
>>>>> >
>>>>> >
>>>>> >
>>>>> > Well, raising the limits is one option but there may be better ones..
>>>>> >
>>>>> > There's an FAQ entry on this:
>>>>> >
>>>>> http://wiki.apache.org/lucene-java/LuceneFAQ#Why_am_I_getting_an_IOException_that_says_.22Too_many_open_files.22.3F
>>>>> >
>>>>> > Take a look at org.apache.lucene.search.SearcherManager "Utility class
>>>>> > to safely share IndexSearcher instances across multiple threads".
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Ian.
>>>>> >
>>>>> >
>>>>> > On Thu, Jan 24, 2013 at 9:18 AM, Rafał Kuć <r....@solr.pl> wrote:
>>>>> >> Hello!
>>>>> >>
>>>>> >> You need to allow the user that is running Lucene to open more files..
>>>>> >> There are plenty of tutorials available on the web. Modify your
>>>>> >> /etc/security/limits.conf and if for example your user is lucene add
>>>>> >> the following (or modify if those already exists):
>>>>> >>
>>>>> >> lucene soft nofile 64000
>>>>> >> lucene hard nofile 64000
>>>>> >>
>>>>> >> Relog and run:
>>>>> >>
>>>>> >> sudo -u lucene -s "ulimit -Sn"
>>>>> >>
>>>>> >> To see if the limits are the ones you set. If they are not, check if
>>>>> >> you don't have pam_limits.so commented in the /etc/pam.d/
>>>>> >>
>>>>> >> --
>>>>> >> Regards,
>>>>> >> Rafał Kuć
>>>>> >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>>>>> >>
>>>>> >>> Here is the log:
>>>>> >>
>>>>> >>> Jan 24, 2013 4:10:33 AM
>>>>> >>> org.apache.tomcat.util.net.AprEndpoint$Acceptor run
>>>>> >>> SEVERE: Socket accept failed
>>>>> >>> org.apache.tomcat.jni.Error: 24: Too many open files
>>>>> >>> at org.apache.tomcat.jni.Socket.accept(Native Method)
>>>>> >>> at
>>>>> >>>
>>>>> org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:990)
>>>>> >>> at java.lang.Thread.run(Thread.java:722)
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>> Too many open files... How to solve it?
>>>>> >>
>>>>> >>
>>>>> >>> On Tue, Jan 22, 2013 at 10:52 PM, Michael McCandless <
>>>>> >>> lucene@mikemccandless.com> wrote:
>>>>> >>
>>>>> >>>> Can you post the full stack trace of the CorruptIndexException?
>>>>> >>>>
>>>>> >>>> Mike McCandless
>>>>> >>>>
>>>>> >>>> http://blog.mikemccandless.com
>>>>> >>>>
>>>>> >>>> On Tue, Jan 22, 2013 at 8:20 AM, Cheng <zh...@gmail.com>
>>>>> wrote:
>>>>> >>>> > Hi,
>>>>> >>>> >
>>>>> >>>> > I run a Lucene application on Tomcat. The app will try to open a
>>>>> Linux
>>>>> >>>> > directory, and sometime returns CorruptIndexException error.
>>>>> >>>> >
>>>>> >>>> > Shortly after I restart Tomcat (nothing else changes), the app can
>>>>> be run
>>>>> >>>> > on the fly. I am using the following statements to open a directory:
>>>>> >>>> >
>>>>> >>>> >
>>>>> >>>> > try {
>>>>> >>>> > searcher = new IndexSearcher(IndexReader.open(new
>>>>> >>>> NIOFSDirectory(new
>>>>> >>>> > File("/home/user/"+ type))));
>>>>> >>>> > } catch (IOException e) {
>>>>> >>>> > throw new Exception("[" + type + "] Cannot open index
>>>>> folder...");
>>>>> >>>> > }
>>>>> >>>> >
>>>>> >>>> > I would like to know how to tackle this problem.
>>>>> >>>> >
>>>>> >>>> > Many thanks!
>>>>> >>>>
>>>>> >>>> ---------------------------------------------------------------------
>>>>> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>> >>>>
>>>>> >>>>
>>>>> >>
>>>>> >>
>>>>> >> ---------------------------------------------------------------------
>>>>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>> >>
>>>>> >
>>>>> > ---------------------------------------------------------------------
>>>>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 回复： 回复： 回复： 回复： 回复： IndexReader.open and CorruptIndexException
Posted by Ian Lea <ia...@gmail.com>.
You didn't tell us until now that you were running an old version of
lucene. Read the 3.5.0 javadocs for SearcherManager: "warmer - An
optional SearcherWarmer. Pass null if you don't require ...".
Or upgrade.
--
Ian.
On Fri, Jan 25, 2013 at 5:18 PM, zhoucheng2008 <zh...@gmail.com> wrote:
> SearcherManager sm = new SearcherManager(dir, new SearcherFactory());
>
>
> new SearchFactory() is not recognized in my lucene. asked to change to SearcherWarmer. Why?
>
>
> I have 3.5
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Michael McCandless"<lu...@mikemccandless.com>;
> 发送时间: 2013年1月26日(星期六) 凌晨0:35
> 收件人: "java-user"<ja...@lucene.apache.org>;
>
> 主题: Re: 回复： 回复： 回复： 回复： IndexReader.open and CorruptIndexException
>
>
>
> That should work.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Fri, Jan 25, 2013 at 11:27 AM, zhoucheng2008 <zh...@gmail.com> wrote:
>> Sorry, I meant this:
>>
>>
>> SearcherManager sm = new SearcherManager(dir, new SearcherFactory());
>>
>>
>>
>>
>> ------------------ 原始邮件 ------------------
>> 发件人: "Ian Lea"<ia...@gmail.com>;
>> 发送时间: 2013年1月26日(星期六) 凌晨0:16
>> 收件人: "java-user"<ja...@lucene.apache.org>;
>>
>> 主题: Re: 回复： 回复： 回复： IndexReader.open and CorruptIndexException
>>
>>
>>
>>> Is SearcherFactory the same as SearcherManager?
>>
>> No.
>>
>>> Ian mentioned a new warmer() solution. Maybe I can try that first.
>>
>> Really? Don't remember that.
>>
>>
>> --
>> Ian.
>>
>>> ------------------ 原始邮件 ------------------
>>> 发件人: "Michael McCandless"<lu...@mikemccandless.com>;
>>> 发送时间: 2013年1月26日(星期六) 凌晨0:10
>>> 收件人: "java-user"<ja...@lucene.apache.org>;
>>>
>>> 主题: Re: 回复： 回复： IndexReader.open and CorruptIndexException
>>>
>>>
>>>
>>> You can pass null for the SearcherFactory ... then SearcherManager
>>> will just do new IndexSearcher(reader) for you.
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>> On Fri, Jan 25, 2013 at 9:30 AM, zhoucheng2008 <zh...@gmail.com> wrote:
>>>> SearcherManager mgr = new SearcherManager(dir, new MySearchWarmer());Can I put null as the second parameter in the SearcherManager constructor?If not, how to create a MySearchWarmer instance? The following explanation doesn't provide sufficient info:
>>>> class MySearchWarmer implements SearchWarmer { @Override public void warm(IndexSearcher searcher) throws IOException { // Run some diverse searches, searching and sorting against all // fields that are used by your application } }
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ------------------ 原始邮件 ------------------
>>>> 发件人: "Michael McCandless"<lu...@mikemccandless..com>;
>>>> 发送时间: 2013年1月25日(星期五) 晚上9:26
>>>> 收件人: "java-user"<ja...@lucene.apache.org>;
>>>>
>>>> 主题: Re: 回复： IndexReader.open and CorruptIndexException
>>>>
>>>>
>>>>
>>>> Maybe here?:
>>>>
>>>> http://blog.mikemccandless.com/2011/11/near-real-time-readers-with-lucenes.html
>>>>
>>>> Mike McCandless
>>>>
>>>> http://blog.mikemccandless.com
>>>>
>>>> On Fri, Jan 25, 2013 at 7:58 AM, Cheng <zh...@gmail.com> wrote:
>>>>> Any example code for this SearcherManager?
>>>>>
>>>>> On Fri, Jan 25, 2013 at 3:59 AM, Ian Lea <ia...@gmail.com> wrote:
>>>>>
>>>>>> There will be one file handle for every currently open file.
>>>>>>
>>>>>> Use SearcherManager and this problem should go away.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ian.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 6:40 PM, zhoucheng2008 <zh...@gmail.com>
>>>>>> wrote:
>>>>>> > What file handlers did you guy refer to?
>>>>>> >
>>>>>> >
>>>>>> > I opened the index directory only. Is this the file handler? Also, how
>>>>>> to safely and effectively close the index directory?
>>>>>> >
>>>>>> >
>>>>>> > I found the link's explanation somewhat self-contradictory. After I read
>>>>>> it, I am confused if I should close the file handlers in the finally block
>>>>>> or not. I am using Java.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > ------------------ 原始邮件 ------------------
>>>>>> > 发件人: "Ian Lea"<ia...@gmail.com>;
>>>>>> > 发送时间: 2013年1月24日(星期四) 下午5:46
>>>>>> > 收件人: "java-user"<ja...@lucene.apache.org>;
>>>>>> >
>>>>>> > 主题: Re: IndexReader.open and CorruptIndexException
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > Well, raising the limits is one option but there may be better ones..
>>>>>> >
>>>>>> > There's an FAQ entry on this:
>>>>>> >
>>>>>> http://wiki.apache.org/lucene-java/LuceneFAQ#Why_am_I_getting_an_IOException_that_says_.22Too_many_open_files.22.3F
>>>>>> >
>>>>>> > Take a look at org.apache.lucene.search.SearcherManager "Utility class
>>>>>> > to safely share IndexSearcher instances across multiple threads".
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Ian.
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Jan 24, 2013 at 9:18 AM, Rafał Kuć <r....@solr.pl> wrote:
>>>>>> >> Hello!
>>>>>> >>
>>>>>> >> You need to allow the user that is running Lucene to open more files..
>>>>>> >> There are plenty of tutorials available on the web. Modify your
>>>>>> >> /etc/security/limits.conf and if for example your user is lucene add
>>>>>> >> the following (or modify if those already exists):
>>>>>> >>
>>>>>> >> lucene soft nofile 64000
>>>>>> >> lucene hard nofile 64000
>>>>>> >>
>>>>>> >> Relog and run:
>>>>>> >>
>>>>>> >> sudo -u lucene -s "ulimit -Sn"
>>>>>> >>
>>>>>> >> To see if the limits are the ones you set. If they are not, check if
>>>>>> >> you don't have pam_limits.so commented in the /etc/pam.d/
>>>>>> >>
>>>>>> >> --
>>>>>> >> Regards,
>>>>>> >> Rafał Kuć
>>>>>> >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>>>>>> >>
>>>>>> >>> Here is the log:
>>>>>> >>
>>>>>> >>> Jan 24, 2013 4:10:33 AM
>>>>>> >>> org.apache.tomcat.util.net.AprEndpoint$Acceptor run
>>>>>> >>> SEVERE: Socket accept failed
>>>>>> >>> org.apache.tomcat.jni.Error: 24: Too many open files
>>>>>> >>> at org.apache.tomcat.jni.Socket.accept(Native Method)
>>>>>> >>> at
>>>>>> >>>
>>>>>> org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:990)
>>>>>> >>> at java.lang.Thread.run(Thread.java:722)
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>> Too many open files... How to solve it?
>>>>>> >>
>>>>>> >>
>>>>>> >>> On Tue, Jan 22, 2013 at 10:52 PM, Michael McCandless <
>>>>>> >>> lucene@mikemccandless.com> wrote:
>>>>>> >>
>>>>>> >>>> Can you post the full stack trace of the CorruptIndexException?
>>>>>> >>>>
>>>>>> >>>> Mike McCandless
>>>>>> >>>>
>>>>>> >>>> http://blog.mikemccandless.com
>>>>>> >>>>
>>>>>> >>>> On Tue, Jan 22, 2013 at 8:20 AM, Cheng <zh...@gmail.com>
>>>>>> wrote:
>>>>>> >>>> > Hi,
>>>>>> >>>> >
>>>>>> >>>> > I run a Lucene application on Tomcat. The app will try to open a
>>>>>> Linux
>>>>>> >>>> > directory, and sometime returns CorruptIndexException error.
>>>>>> >>>> >
>>>>>> >>>> > Shortly after I restart Tomcat (nothing else changes), the app can
>>>>>> be run
>>>>>> >>>> > on the fly. I am using the following statements to open a directory:
>>>>>> >>>> >
>>>>>> >>>> >
>>>>>> >>>> > try {
>>>>>> >>>> > searcher = new IndexSearcher(IndexReader.open(new
>>>>>> >>>> NIOFSDirectory(new
>>>>>> >>>> > File("/home/user/"+ type))));
>>>>>> >>>> > } catch (IOException e) {
>>>>>> >>>> > throw new Exception("[" + type + "] Cannot open index
>>>>>> folder...");
>>>>>> >>>> > }
>>>>>> >>>> >
>>>>>> >>>> > I would like to know how to tackle this problem.
>>>>>> >>>> >
>>>>>> >>>> > Many thanks!
>>>>>> >>>>
>>>>>> >>>> ---------------------------------------------------------------------
>>>>>> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> >>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>
>>>>>> >>
>>>>>> >> ---------------------------------------------------------------------
>>>>>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>> >>
>>>>>> >
>>>>>> > ---------------------------------------------------------------------
>>>>>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>
>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org